Finally, I found:
Removal of lock_kernel in fs/fcntl.c causes the strange performance of
2.4.0-test9.
The removal causes following negative scalability on Apache-1.3.9:
* 8-way performance dropped to 60% of 4-way performance.
* Adding lock_kernel() gains 2.4x performance on 8-way.
This suggests some design malfunction exist in the fs-code.
The lock_kernel() is removed in test9, as shown in below, then the
strange behavior appeared.
linux-2.4.0-test8/fs/fcntl.c:
asmlinkage long sys_fcntl(unsigned int fd, unsigned int cmd, unsigned long arg)
{
struct file * filp;
long err = -EBADF;
filp = fget(fd);
if (!filp)
goto out;
--> lock_kernel();
err = do_fcntl(fd, cmd, arg, filp);
--> unlock_kernel();
fput(filp);
out:
return err;
}
Adding the lock_kernel()/unlock_kernel() to test9:fs/fcntl.c,
The performance is restored,
The number of task switch is reduced, and
Positive scalability is observed.
The lock region may be narrowed to around call of posix_lock_file()
in fcntl_setlk() (fs/locks.c).
I usually prefer removal of kernel_lock, but at this time,
the removal severy struck the performance.
Please give me suggestions..
[email protected] writes:
> [email protected] writes:
> > Rik van Riel writes:
> > > On Wed, 25 Oct 2000 [email protected] wrote:
> > > > I found very odd performance behavior of 2.4.0-test9 on a large SMP
> > > > server, and I want some clues to investigate it.
> > > >
> > > > 1) At the 8 cpu configuration, test9 shows extremely inferior
> > > > performance.
> > > > 2) on test8, 8-cpu configuration shows about 2/3 performance of 4-cpu.
> > > ^^^^^ test9 ??
>
> IMHO, the modification of file-system code causes the weird
> performance.
>
> Most of processes are slept at:
> posix_lock_file()->locks_block_on()->interruptible_sleep_on_locks()
>
> We revert two of test9 files (fs/fcntl.c fs/flock.c), to the previous
> version, the performance problem disappeared and it becomes to the
> same level as test8.
>
> To narrow the problem, we measured performance of 3 configuration:
> 1) test9 with test8 fs/fcntl.c, test8 fs/flock.c
> 2) test9 with test8 fs/fcntl.c
> 3) test9 with test8 fs/flock.c
>
> Only 3) shows the problem, so the main problem reside in fcntl.c (not
> in flock.c).
>
> So it seems:
> the web-server, apache-1.3.9 in the redhat-6.1, issues lots of fcntl
> to the file and those fcntls collide each other, and the processes
> are blocked.
>
>
> What has happend to fcntl.c?
>
> --
> Computer Systems Laboratory, Fujitsu Labs.
> [email protected]