From: "Chuck Lever" Subject: Re: [NFS] [ltc-perf] draft of nfs event hook Date: Thu, 27 Jul 2006 11:29:39 -0400 Message-ID: <76bd70e30607270829q500711c8k8e7dd6ba9a9b6dfe@mail.gmail.com> References: <44C8C631.40003@cn.ibm.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Cc: "systemtap@sourceware.org" , nfs@lists.sourceforge.net Return-path: To: "Li Guanglei" In-Reply-To: <44C8C631.40003@cn.ibm.com> List-Unsubscribe: List-Subscribe: List-Post: List-Help: , Sender: systemtap-owner@sourceware.org List-ID: On 7/27/06, Li Guanglei wrote: > When we started working on NFS trace hooks, we realized it is not > an easy task. Although we use NFS in daily work but we don't have much > knowledge about the NFS protocol details and its implementation inside > the Kernel. So I divided the work into two steps. At the first step I > need get a list of trace points. And at the second step I need to make > sure what trace data is available for each trace hook. In a short, the > trace data available for each hook will be derived from the arguments > of the kernel functions being probed. > > We read through the Kernel source code and chose some functions to > be instrumented. We will trace the entry of these functions and if > necessary, the return of them will also be traced. The following is > the list of these functions, please take a review: Have you done this with a local file system? I assume yes, and that you just described the general approach you have taken with other file systems. I think getting the same kind of data and trace points from the NFS client as you added to local file systems would be good. Capturing VFS and address space entry points is definitely useful and is similar to local file systems. At the bottom of the NFS client is the RPC client, and it acts just like the block I/O layer does for local file systems. Would you consider adding trace points in the LKET for the RPC client and server? > ==================== Client Side ========================== > > <1> nfs directory operations > > All functions from nfs_dir_operations: > > const struct file_operations nfs_dir_operations = { > .llseek = nfs_llseek_dir, > .read = generic_read_dir, > .readdir = nfs_readdir, > .open = nfs_opendir, > .release = nfs_release, > .fsync = nfs_fsync_dir, > }; > > <2> nfs file operations > > All functions from nfs_file_operations: > > const struct file_operations nfs_file_operations = { > .llseek = nfs_file_llseek, > .read = do_sync_read, > .write = do_sync_write, > .aio_read = nfs_file_read, > .aio_write = nfs_file_write, > .mmap = nfs_file_mmap, > .open = nfs_file_open, > .flush = nfs_file_flush, > .release = nfs_file_release, > .fsync = nfs_fsync, > .lock = nfs_lock, > .flock = nfs_flock, > .sendfile = nfs_file_sendfile, > .check_flags = nfs_check_flags, > }; > > <3> nfs address space operations: > All functions from nfs_file_aops: > > struct address_space_operations nfs_file_aops = { > .readpage = nfs_readpage, > .readpages = nfs_readpages, > .set_page_dirty = __set_page_dirty_nobuffers, > .writepage = nfs_writepage, > .writepages = nfs_writepages, > .prepare_write = nfs_prepare_write, > .commit_write = nfs_commit_write, > .invalidatepage = nfs_invalidate_page, > .releasepage = nfs_release_page, > #ifdef CONFIG_NFS_DIRECTIO > .direct_IO = nfs_direct_IO, > #endif > }; > > <4> NFS RPC procedures: > > All functions from nfs_v[2,3,4]_clientops: > I only list the nfs_v3 rpc procedures: > struct nfs_rpc_ops nfs_v3_clientops = { > .version = 3, /* protocol version */ > .dentry_ops = &nfs_dentry_operations, > .dir_inode_ops = &nfs3_dir_inode_operations, > .file_inode_ops = &nfs3_file_inode_operations, > .getroot = nfs3_proc_get_root, > .getattr = nfs3_proc_getattr, > .setattr = nfs3_proc_setattr, > .lookup = nfs3_proc_lookup, > .access = nfs3_proc_access, > .readlink = nfs3_proc_readlink, > .read = nfs3_proc_read, > .write = nfs3_proc_write, > .commit = nfs3_proc_commit, > .create = nfs3_proc_create, > .remove = nfs3_proc_remove, > .unlink_setup = nfs3_proc_unlink_setup, > .unlink_done = nfs3_proc_unlink_done, > .rename = nfs3_proc_rename, > .link = nfs3_proc_link, > .symlink = nfs3_proc_symlink, > .mkdir = nfs3_proc_mkdir, > .rmdir = nfs3_proc_rmdir, > .readdir = nfs3_proc_readdir, > .mknod = nfs3_proc_mknod, > .statfs = nfs3_proc_statfs, > .fsinfo = nfs3_proc_fsinfo, > .pathconf = nfs3_proc_pathconf, > .decode_dirent = nfs3_decode_dirent, > .read_setup = nfs3_proc_read_setup, > .read_done = nfs3_read_done, > .write_setup = nfs3_proc_write_setup, > .write_done = nfs3_write_done, > .commit_setup = nfs3_proc_commit_setup, > .commit_done = nfs3_commit_done, > .file_open = nfs_open, > .file_release = nfs_release, > .lock = nfs3_proc_lock, > .clear_acl_cache = nfs3_forget_cached_acls, > }; > > The LKET already has syscall and iosyscall trace hooks. So with the > above trace hooks, LKET could trace different layer of NFS operations: > --> Syscall > --> struct file_operations > --> struct address_space_operations > --> struct nfs_rpc_ops > > ======================= Server Side ============================= > > <1> nfsd_dispatch > This is the NFS dispatching function sit on top of RPC. > > <2> NFS RPC procedures: > > For NFSv4, it will be nfsd4_proc_compound > > For NFSv2, NFSv3, it will be the functions from nfsd_procedures[2,3] > > Here is a list for NFSv3, NFSv2 are almost the same: > nfsd3_proc_null, > nfsd3_proc_getattr, > nfsd3_proc_setattr, > nfsd3_proc_lookup, > nfsd3_proc_access, > nfsd3_proc_readlink, > nfsd3_proc_read, > nfsd3_proc_write, > nfsd3_proc_create, > nfsd3_proc_mkdir, > nfsd3_proc_symlink, > nfsd3_proc_mknod, > nfsd3_proc_remove, > nfsd3_proc_rmdir, > nfsd3_proc_rename, > nfsd3_proc_link, > nfsd3_proc_readdir, > nfsd3_proc_readdirplus,readdirplus, > nfsd3_proc_fsstat, > nfsd3_proc_fsinfo, > nfsd3_proc_pathconf, > nfsd3_proc_commit, > > <3> NFSD file VFS operations > > The functions nfsd_xxx from "fs/nfsd/vfs.c" > > With the above server side trace hooks, LKET could trace NFS > operations at different layer: > > nfsd_dispatch --> > --> NFS RPC Procedures > --> NFS VFS file operations > > > What I didn't list about NFS operations includes authentication, > NFSv4 callback and RPC(I prefer to use a separate set of trace hooks > for RPC). I am not sure if these operations are also required to be > traced. If I missed some important functions or I listed some > redundant functions, please feel free to let me know. Any comments > will be highly appreciated. > > Thanks. > > The following is from Li Xuepeng posted on nfs@lists.sourceforge.net > which involved some implementations details and its trace point lists > is a subset of the above. > > - Guanglei > > Xue Peng Li ??: > > Hi folks, > > > > I am working on NFS trace hooks for SystemTap/LKET. These trace > > hooks could be used for performance analyzing which will trace both > > NFS client and server side activities. > > > > At the first step I need make sure that the trace hooks I defined > > are appropriate and every trace hook probes the right places inside > > the Kernel. So I will be appreciated if you could help me review the > > following trace hooks. > > > > > > Thanks > > > > ======================== NFS Client Side Trace Hooks ================= > > > > The following event hooks are used to trace nfs client activities. > > These event hooks are divided into two groups. Probe Point and > > Description is given for each event hook. > > > > Group1: > > It contains 15 event hooks, which are used to probe Client-side > > NFS procedures. > > --------------------------------------------------------------------- > > addevent.nfs.proc.read_setup > > Probe Point: > > nfs_proc_read_setup,nfs3_proc_read_setup, nfs4_proc_read_setup > > Description: > > Setup a rpc task to prepare for reading > > --------------------------------------------------------------------- > > addevent.nfs.proc.read_done > > Probe Point: > > nfs_proc_read_done,nfs3_proc_read_done, > > nfs4_proc_read_done > > Description: > > Fires when receive a read reply from server,it is used to > > refresh the inode on client > > --------------------------------------------------------------------- > > addevnet.nfs.proc.read > > Probe Point: > > nfs_proc_read,nfs3_proc_read,nfs4_proc_read > > Description: > > Send a read operation to server,and refresh local inode after > > receive reply from server > > --------------------------------------------------------------------- > > addevent.nfs.proc.write_setup > > Probe Point: > > nfs_proc_write_setup,nfs3_proc_write_setup,nfs4_proc_write_setup > > Description: > > --------------------------------------------------------------------- > > addevent.nfs.proc.write > > Probe Point: > > nfs_proc_write,nfs3_proc_write,nfs4_proc_write > > Description: > > Send a write operation to server > > --------------------------------------------------------------------- > > addevent.nfs.proc.write_done > > Probe Point: > > nfs_write_done,nfs3_write_done,nfs4_write_done > > Description: > > Fires when receive a write reply from server,it is used to > > refresh the inode on client > > --------------------------------------------------------------------- > > addevent.nfs.proc.open > > Probe Point: > > nfs_open > > Description: > > Allocate file read/write context information > > --------------------------------------------------------------------- > > addevent.nfs.proc.release > > Probe Point: > > nfs_release > > Description: > > Release file read/write context information > > --------------------------------------------------------------------- > > addevent.nfs.proc.create > > Probe Point: > > nfs_create > > Description: > > Create a new file or dir on server > > _____________________________________________________________________ > > > > Group2: > > This group includes the event hooks which probe NFS address space > > operation related function.All the functions are common in NFSV2, > > NFSV3,NFSV4. > > --------------------------------------------------------------------- > > addevent.nfs.aops.readpage > > Probe Point: > > nfs_readpage > > Description : > > Read the page ,only fires when a previous async read operation > > failed > > --------------------------------------------------------------------- > > addevent.nfs.aops.readpages > > Probe Point: > > nfs_readpages > > Description: > > Fires when in readahead way,read several pages once > > --------------------------------------------------------------------- > > addevent.nfs.aops.writepage > > Probe Point: > > nfs_writepage > > Description: > > Write an mapped page to the server > > --------------------------------------------------------------------- > > addevent.nfs.aops.writepages > > Probe Point: > > nfs_writepages > > Description: > > Write several dirty pages to the serve once > > --------------------------------------------------------------------- > > addevent.nfs.aops.prepare_write > > Probe Point: > > prepare_write > > Description: > > Prepare a page for writing. Look for a request corresponding > > to the page. If there is one, and it belongs to another aops, > > we flush it out before we try to copy anything into the page. > > Also do the same if we find a request from an existing > > dropped page. > > --------------------------------------------------------------------- > > addevent.nfs.aops.commit_write > > Probe Point: > > nfs_commit_write > > Description : > > Update and possibly write a cached page of an NFS aops > > _____________________________________________________________________ > > > > > > ====================== NFS Server Side Trace Hooks ================== > > > > The following event hooks are used to traced nfs server activities. > > The event hooks are divided into three group. > > > > Group1: > > It contains one event hook,which probes nfsd_dispatch > > --------------------------------------------------------------------- > > addevent.nfsd.dispatch > > Probe Point: > > nfsd_dispatch > > Description: > > Decode the arguments received from client,call the procedure > > handler,encode the result > > ______________________________________________________________________ > > Group2: > > It contains three event hooks.The functions probed will be called > > by related procedure handler. All the functions are common in NFSV2, > > NFSV3,NFSV4 > > --------------------------------------------------------------------- > > addevent.nfsd.read > > Probe Point: > > nfsd_read > > Description: > > It does the "real" work of read > > --------------------------------------------------------------------- > > addevent.nfsd.write > > Probe Point: > > nfsd_write > > Description: > > It does the "real " work of write > > --------------------------------------------------------------------- > > addevent.nfsd.open > > Probe Point: > > nfsd_open > > Description: > > Open an existing file or directory. > > --------------------------------------------------------------------- > > addevent.nfsd.close > > Probe Point: > > nfsd_close > > Description: > > Close an existing file or directory > > _____________________________________________________________________ > > Group3: > > It contains eight event hooks,which probe procedure handlers. > > --------------------------------------------------------------------- > > addevent.nfsd.proc2.read > > Probe Point: > > nfsd_proc_read > > Description: > > Read data from file (NFSV2) > > --------------------------------------------------------------------- > > addevent.nfsd.proc3.read > > Probe Point: > > nfsd3_proc_read > > Description: > > Read data from file (NFSV3) > > --------------------------------------------------------------------- > > addevent.nfsd.proc4.read > > Probe Point: > > nfsd4_read > > Description: > > Check stateid and prepare for reading > > --------------------------------------------------------------------- > > addevent.nfsd.proc2.write > > Probe Point: > > nfsd_proc_write > > Description: > > Write data to file (NFSV2) > > --------------------------------------------------------------------- > > addevent.nfsd.pro3.write > > Probe Point: > > nfsd3_proc_write > > Description: > > Write data to file (NFSV3) > > --------------------------------------------------------------------- > > addevent.nfsd.proc4.write > > Probe Point: > > nfsd4_write > > Description: > > Check stateid and write data to file > > --------------------------------------------------------------------- > > addevent.nfsd.proc4.open > > Probe Point: > > nfsd4_open > > Description: > > Check stateid and open file > > --------------------------------------------------------------------- > > addevent.nfsd.proc4.compound > > Probe Point: > > nfsd4_proc_compound > > Description: > > Call different procedures according to client request > > > > > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > ltc-perf mailing list > > ltc-perf@linux.ibm.com > > http://linux.ibm.com/mailman/listinfo/ltc-perf > > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > NFS maillist - NFS@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs > -- "We who cut mere stones must always be envisioning cathedrals" -- Quarry worker's creed