Hello,
On Mon, Jun 14, 2021 at 11:23:02AM -0400, Waiman Long wrote:
> The current container management system is able to create the illusion
> that applications running within a container have limited resources and
> devices available for their use. However, one thing that is hard to hide
> is the number of CPUs available in the system. In fact, the container
> developers are asking for the kernel to provide such capability.
>
> There are two places where cpu information are available for the
> applications to see - /proc/cpuinfo and /sys/devices/system/cpu sysfs
> directory.
>
> This patchset introduces a new sysctl parameter cpuset_bound_cpuinfo
> which, when set, will limit the amount of information disclosed by
> /proc/cpuinfo and /sys/devices/system/cpu.
The goal of cgroup has never been masquerading system information so that
applications can pretend that they own the whole system and the proposed
solution requires application changes anyway. The information being provided
is useful but please do so within the usual cgroup interface - e.g.
cpuset.stat. The applications (or libraries) that want to determine its
confined CPU availability can locate the file through /proc/self/cgroup.
Thanks.
--
tejun
On 6/14/21 4:43 PM, Tejun Heo wrote:
> Hello,
>
> On Mon, Jun 14, 2021 at 11:23:02AM -0400, Waiman Long wrote:
>> The current container management system is able to create the illusion
>> that applications running within a container have limited resources and
>> devices available for their use. However, one thing that is hard to hide
>> is the number of CPUs available in the system. In fact, the container
>> developers are asking for the kernel to provide such capability.
>>
>> There are two places where cpu information are available for the
>> applications to see - /proc/cpuinfo and /sys/devices/system/cpu sysfs
>> directory.
>>
>> This patchset introduces a new sysctl parameter cpuset_bound_cpuinfo
>> which, when set, will limit the amount of information disclosed by
>> /proc/cpuinfo and /sys/devices/system/cpu.
> The goal of cgroup has never been masquerading system information so that
> applications can pretend that they own the whole system and the proposed
> solution requires application changes anyway. The information being provided
> is useful but please do so within the usual cgroup interface - e.g.
> cpuset.stat. The applications (or libraries) that want to determine its
> confined CPU availability can locate the file through /proc/self/cgroup.
Thanks for your comment. I understand your point making change via
cgroup interface files. However, this is not what the customers are
asking for. They are using tools that look at /proc/cpuinfo and the
sysfs files. It is a much bigger effort to make all those tools look at
a new cgroup file interface instead. It can be more efficiently done at
the kernel level.
Anyway, I am OK if the consensus is that it is not a kernel problem and
have to be handled in userspace.
BTW, do you have any comment on another cpuset patch that I sent a week
earlier?
https://lore.kernel.org/lkml/[email protected]/
I am looking forward for your feedback.
Cheers,
Longman
On Mon, Jun 14, 2021 at 04:43:28PM -0400, Tejun Heo wrote:
> Hello,
>
> On Mon, Jun 14, 2021 at 11:23:02AM -0400, Waiman Long wrote:
> > The current container management system is able to create the illusion
> > that applications running within a container have limited resources and
> > devices available for their use. However, one thing that is hard to hide
> > is the number of CPUs available in the system. In fact, the container
> > developers are asking for the kernel to provide such capability.
> >
> > There are two places where cpu information are available for the
> > applications to see - /proc/cpuinfo and /sys/devices/system/cpu sysfs
> > directory.
> >
> > This patchset introduces a new sysctl parameter cpuset_bound_cpuinfo
> > which, when set, will limit the amount of information disclosed by
> > /proc/cpuinfo and /sys/devices/system/cpu.
>
> The goal of cgroup has never been masquerading system information so that
> applications can pretend that they own the whole system and the proposed
> solution requires application changes anyway. The information being provided
> is useful but please do so within the usual cgroup interface - e.g.
> cpuset.stat. The applications (or libraries) that want to determine its
> confined CPU availability can locate the file through /proc/self/cgroup.
Fyi, there's another concurrent push going on to provide a new file
/proc/self/meminfo that is a subset of /proc/meminfo (cf. [1]) and
virtualizes based on cgroups as well.
But there it's a new file not virtualizing exisiting files and
directories so there things seem to be out of sync between these groups
at the same company.
In any case I would like to point out that this has a complete solution
in userspace. We have had this problem of providing virtualized
information to containers since they started existing. So we created
LXCFS in 2014 (cf. [2]) a tiny fuse fileystem to provide a virtualized
view based on cgroups and other information for containers.
The two patchsets seems like they're on the way trying to move 1:1 what
we're already doing in userspace into the kernel. LXCFS is quite well
known and widely used so it's suprising to not see it mentioned at all.
And the container people will want more then just the cpu and meminfo
stuff sooner or later. Just look at what we currently virtualize:
/proc/cpuinfo
/proc/diskstats
/proc/meminfo
/proc/stat
/proc/swaps
/proc/uptime
/proc/slabinfo
/sys/devices/system/cpu
## So for example /proc/cpuinfo
#### Host
brauner@wittgenstein|~
> grep ^processor /proc/cpuinfo
processor : 0
processor : 1
processor : 2
processor : 3
processor : 4
processor : 5
processor : 6
processor : 7
#### Container
brauner@wittgenstein|~
> lxc exec f1 -- grep ^processor /proc/cpuinfo
processor : 0
processor : 1
## and for /sys/devices/system/cpu
#### Host
brauner@wittgenstein|~
> ls -al /sys/devices/system/cpu/ | grep cpu[[:digit:]]
drwxr-xr-x 10 root root 0 Jun 14 21:22 cpu0
drwxr-xr-x 10 root root 0 Jun 14 21:22 cpu1
drwxr-xr-x 10 root root 0 Jun 14 21:22 cpu2
drwxr-xr-x 10 root root 0 Jun 14 21:22 cpu3
drwxr-xr-x 10 root root 0 Jun 14 21:22 cpu4
drwxr-xr-x 10 root root 0 Jun 14 21:22 cpu5
drwxr-xr-x 10 root root 0 Jun 14 21:22 cpu6
drwxr-xr-x 10 root root 0 Jun 14 21:22 cpu7
#### Container
brauner@wittgenstein|~
> lxc exec f1 -- ls -al /sys/devices/system/cpu/ | grep cpu[[:digit:]]
drwxr-xr-x 2 nobody nogroup 0 Jun 15 09:07 cpu3
drwxr-xr-x 2 nobody nogroup 0 Jun 15 09:07 cpu4
We have a wide variety of users from various distros.
[1]: https://lore.kernel.org/containers/[email protected]
[2]: https://github.com/lxc/lxcfs
Hello, Waiman.
On Mon, Jun 14, 2021 at 10:53:53PM -0400, Waiman Long wrote:
> Thanks for your comment. I understand your point making change via cgroup
> interface files. However, this is not what the customers are asking for.
It's not like we can always follow what specific customers request. If there
are actual use-cases that can't be achieved with the existing interfaces and
features, we can look into how to provide those but making interface
decisions based on specific customer requests tends to lead to long term
pains.
> They are using tools that look at /proc/cpuinfo and the sysfs files. It is a
> much bigger effort to make all those tools look at a new cgroup file
> interface instead. It can be more efficiently done at the kernel level.
Short term, sure, it sure is more painful to adapt, but I don't think longer
term solution lies in the kernel trying to masquerage existing sytsem-wide
information interfaces. e.g. cpuset is one thing but what are we gonna do
about weight control or work-conserving memory controls? Pro-rate cpu count
and available memory?
> Anyway, I am OK if the consensus is that it is not a kernel problem and have
> to be handled in userspace.
I'd be happy to provide more information from kernel side as necessary but
the approach taken here doesn't seem generic or scalable at all.
> BTW, do you have any comment on another cpuset patch that I sent a week
> earlier?
>
> https://lore.kernel.org/lkml/[email protected]/
>
> I am looking forward for your feedback.
Sorry about the delay. Will take a look later today.
Thanks.
--
tejun