Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752952AbYKQSud (ORCPT ); Mon, 17 Nov 2008 13:50:33 -0500 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1751657AbYKQSuY (ORCPT ); Mon, 17 Nov 2008 13:50:24 -0500 Received: from mx2.mail.elte.hu ([157.181.151.9]:51157 "EHLO mx2.mail.elte.hu" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751479AbYKQSuS (ORCPT ); Mon, 17 Nov 2008 13:50:18 -0500 Date: Mon, 17 Nov 2008 19:49:51 +0100 From: Ingo Molnar To: Linus Torvalds Cc: Eric Dumazet , David Miller , rjw@sisk.pl, linux-kernel@vger.kernel.org, kernel-testers@vger.kernel.org, cl@linux-foundation.org, efault@gmx.de, a.p.zijlstra@chello.nl, Stephen Hemminger Subject: Re: [Bug #11308] tbench regression on each kernel release from 2.6.22 -> 2.6.28 Message-ID: <20081117184951.GA5585@elte.hu> References: <20081117.011403.06989342.davem@davemloft.net> <20081117110119.GL28786@elte.hu> <4921539B.2000002@cosmosbay.com> <20081117161135.GE12081@elte.hu> <49219D36.5020801@cosmosbay.com> <20081117170844.GJ12081@elte.hu> <20081117172549.GA27974@elte.hu> <4921AAD6.3010603@cosmosbay.com> <20081117182320.GA26844@elte.hu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20081117182320.GA26844@elte.hu> User-Agent: Mutt/1.5.18 (2008-05-17) X-ELTE-VirusStatus: clean X-ELTE-SpamScore: -1.5 X-ELTE-SpamLevel: X-ELTE-SpamCheck: no X-ELTE-SpamVersion: ELTE 2.0 X-ELTE-SpamCheck-Details: score=-1.5 required=5.9 tests=BAYES_00,DNS_FROM_SECURITYSAGE autolearn=no SpamAssassin version=3.2.3 -1.5 BAYES_00 BODY: Bayesian spam probability is 0 to 1% [score: 0.0000] 0.0 DNS_FROM_SECURITYSAGE RBL: Envelope sender in blackholes.securitysage.com Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 38116 Lines: 1365 * Ingo Molnar wrote: 4> The place for the sock_rfree() hit looks a bit weird, and i'll > investigate it now a bit more to place the real overhead point > properly. (i already mapped the test-bit overhead: that comes from > napi_disable_pending()) ok, here's a new set of profiles. (again for tbench 64-thread on a 16-way box, with v2.6.28-rc5-19-ge14c8bf and with the kernel config i posted before.) Here are the per major subsystem percentages: NET overhead ( 5786945/10096751): 57.31% security overhead ( 925933/10096751): 9.17% usercopy overhead ( 837887/10096751): 8.30% sched overhead ( 753662/10096751): 7.46% syscall overhead ( 268809/10096751): 2.66% IRQ overhead ( 266500/10096751): 2.64% slab overhead ( 180258/10096751): 1.79% timer overhead ( 92986/10096751): 0.92% pagealloc overhead ( 87381/10096751): 0.87% VFS overhead ( 53295/10096751): 0.53% PID overhead ( 44469/10096751): 0.44% pagecache overhead ( 33452/10096751): 0.33% gtod overhead ( 11064/10096751): 0.11% IDLE overhead ( 0/10096751): 0.00% --------------------------------------------------------- left ( 753878/10096751): 7.47% The breakdown is very similar to what i sent before, within noise. [ 'left' is random overhead from all around the place - i categorized the 500 most expensive functions in the profile per subsystem. I stopped short of doing it for all 1300+ functions: it's rather laborous manual work even with hefty use of regex patterns. It's also less meaningful in practice: the trend in the first 500 functions is present in the remaining 800 functions as well. I watched the breakdown evolve as i increased the coverage - in practice it is the first 100 functions that matter - it just doesnt change after that. ] The readprofile output below seems structured in a more useful way now - i tweaked compiler options to have the profiler hits spread out in a more meaningful way. I collected 10 million NMI profiler hits, and normalized the readprofile output up to 100%. [ I'll post per function analysis as i complete them, as a reply to this mail. ] Ingo 100.000000 total ................ 7.253355 copy_user_generic_string 3.934833 avc_has_perm_noaudit 3.356152 ip_queue_xmit 3.038025 skb_release_data 2.118525 skb_release_head_state 1.997533 tcp_ack 1.833688 tcp_recvmsg 1.717771 eth_type_trans 1.673249 __inet_lookup_established 1.508888 system_call 1.469183 tcp_current_mss 1.431553 tcp_transmit_skb 1.385125 tcp_sendmsg 1.327643 tcp_v4_rcv 1.292328 nf_hook_thresh 1.203205 schedule 1.059501 nf_hook_slow 1.027373 constant_test_bit 0.945183 sock_rfree 0.922748 __switch_to 0.911605 netif_rx 0.876270 register_gifconf 0.788200 ip_local_deliver_finish 0.781467 dev_queue_xmit 0.766530 constant_test_bit 0.758208 _local_bh_enable_ip 0.747184 load_cr3 0.704341 memset_c 0.671260 sysret_check 0.651845 ip_finish_output2 0.620204 audit_free_names 0.617781 audit_syscall_exit 0.615149 skb_copy_datagram_iovec 0.613848 selinux_socket_sock_rcv_skb 0.606995 constant_test_bit 0.593936 __tcp_push_pending_frames 0.592198 tcp_cleanup_rbuf 0.574093 ip_rcv 0.567886 netif_receive_skb 0.563377 get_page_from_freelist 0.557657 tcp_event_data_recv 0.539274 ip_local_deliver 0.534130 sys_recvfrom 0.512321 __tcp_select_window 0.498427 tcp_rcv_established 0.494862 sys_sendto 0.487473 audit_syscall_entry 0.478495 sched_clock_cpu 0.474861 kfree 0.466310 tcp_established_options 0.461384 net_rx_action 0.447162 __mod_timer 0.442078 ip_rcv_finish 0.441631 find_pid_ns 0.441124 sk_wait_data 0.423943 __sock_recvmsg 0.422126 selinux_parse_skb 0.417975 __napi_schedule 0.414082 __do_softirq 0.403604 task_rq_lock 0.380792 nf_iterate 0.377614 select_task_rq_fair 0.374973 sock_sendmsg 0.374635 kmem_cache_alloc_node 0.368775 avc_has_perm 0.368706 local_bh_disable 0.361834 release_sock 0.346400 sock_common_recvmsg 0.342825 skb_clone 0.338704 __alloc_skb 0.326488 do_softirq 0.323410 lock_sock_nested 0.322129 __copy_skb_header 0.316835 put_page 0.310966 selinux_ip_postroute 0.306229 sel_netport_sid 0.299863 try_to_wake_up 0.296288 process_backlog 0.294818 __inet_lookup 0.294778 thread_return 0.293219 cfs_rq_of 0.292315 internal_add_timer 0.292305 tcp_rcv_space_adjust 0.281053 constant_test_bit 0.278779 local_bh_enable 0.272910 *unknown* 0.269593 schedule_timeout 0.261846 tcp_v4_md5_lookup 0.260992 __ip_local_out 0.255868 __enqueue_entity 0.253931 avc_audit 0.252004 finish_task_switch 0.249263 audit_get_context 0.248290 sockfd_lookup_light 0.247416 virt_to_head_page 0.244149 tcp_options_write 0.243603 memcpy_toiovec 0.243434 sock_recvmsg 0.242599 call_softirq 0.242391 __unlazy_fpu 0.236412 fput_light 0.235628 ret_from_sys_call 0.234933 sk_reset_timer 0.228358 math_state_restore 0.227117 socket_has_perm 0.223492 virt_to_cache 0.219063 __cache_free 0.216401 update_curr 0.216232 tcp_v4_send_check 0.213978 audit_free_aux 0.213223 tcp_v4_do_rcv 0.212975 __kfree_skb 0.211137 dev_hard_start_xmit 0.209052 tcp_rtt_estimator 0.207999 netif_needs_gso 0.207662 __update_sched_clock 0.207284 rb_erase 0.204861 enqueue_task_fair 0.203490 skb_release_all 0.203252 tcp_send_delayed_ack 0.203232 inet_ehashfn 0.199846 sel_netport_find 0.195396 system_call_after_swapgs 0.186756 lock_timer_base 0.186687 pick_next_task_fair 0.183986 mod_timer 0.182982 loopback_xmit 0.182605 native_read_tsc 0.181195 skb_set_owner_r 0.179248 switch_mm 0.175584 set_next_entity 0.173329 raw_local_deliver 0.171641 sys_kill 0.164510 dequeue_task_fair 0.161938 clear_bit 0.160528 sock_def_readable 0.157628 __tcp_ack_snd_check 0.156893 skb_can_coalesce 0.156556 tcp_snd_wnd_test 0.155662 ip_output 0.150627 sk_stream_alloc_skb 0.150219 cpu_sdc 0.149425 sysret_careful 0.148760 tcp_data_snd_check 0.147816 auditsys 0.147419 pskb_may_pull 0.147151 fget_light 0.143774 tcp_cwnd_test 0.143029 rb_insert_color 0.142265 __wake_up 0.141808 tcp_bound_to_half_wnd 0.138600 __sk_dst_check 0.138431 free_hot_cold_page 0.137954 unroll_tree_refs 0.137080 __skb_unlink 0.135124 __sock_sendmsg 0.135064 get_pageblock_flags_group 0.132701 kmem_cache_free 0.128152 bictcp_cong_avoid 0.127874 __napi_complete 0.127527 ____cache_alloc 0.127368 tcp_is_cwnd_limited 0.127278 find_vpid 0.126941 constant_test_bit 0.126504 sk_mem_charge 0.126255 __alloc_pages_internal 0.125977 dst_release 0.125521 hash_64 0.124895 put_prev_task_fair 0.123802 netlbl_enabled 0.122829 sched_clock 0.122640 skb_push 0.122035 __phys_addr 0.121161 dput 0.120515 tcp_prequeue_process 0.118916 __skb_dequeue 0.117715 selinux_socket_sendmsg 0.117536 __inc_zone_state 0.115907 sk_wake_async 0.113504 selinux_ipv4_output 0.113017 sel_netif_sid 0.112431 skb_reset_network_header 0.111170 check_preempt_wakeup 0.111061 bictcp_acked 0.110882 sel_netnode_find 0.109978 update_min_vruntime 0.109889 resched_task 0.109879 current_kernel_time 0.109432 tcp_checksum_complete_user 0.107476 ip_dont_fragment 0.107386 sysret_audit 0.106979 inet_csk_reset_xmit_timer 0.106006 skb_entail 0.105777 sysret_signal 0.105420 avc_hash 0.105251 __skb_clone 0.105211 tcp_init_tso_segs 0.103523 __dequeue_entity 0.101715 PageLRU 0.101378 tcp_parse_aligned_timestamp 0.101219 __xchg 0.100544 constant_test_bit 0.097991 __kmalloc 0.097584 test_tsk_thread_flag 0.097475 autoremove_wake_function 0.095747 selinux_task_kill 0.094416 get_page 0.093353 dequeue_task 0.092728 __local_bh_disable 0.091943 selinux_netlbl_sock_rcv_skb 0.091655 path_put 0.090970 skb_headroom 0.090950 PageTail 0.090642 dst_destroy 0.090523 netpoll_rx 0.089589 skb_header_pointer 0.085935 security_socket_recvmsg 0.084008 alloc_pages_current 0.083184 compare_ether_addr 0.082479 rb_next 0.082439 sk_wmem_schedule 0.081635 next_zones_zonelist 0.080135 tcp_cwnd_validate 0.079877 tcp_event_new_data_sent 0.079817 fcheck_files 0.079082 ip_skb_dst_mtu 0.078804 ip_finish_output 0.078278 wakeup_preempt_entity 0.077026 sel_netif_find 0.076788 __skb_queue_tail 0.076570 sock_flag 0.076520 tcp_win_from_space 0.076510 zone_watermark_ok 0.076282 sel_netnode_sid 0.076162 policy_zonelist 0.074732 __wake_up_common 0.074613 compound_head 0.074593 task_has_perm 0.073243 __find_general_cachep 0.073064 tcp_push 0.072925 skb_cloned 0.072309 pskb_may_pull 0.071852 TCP_ECN_check_ce 0.071495 cap_task_to_inode 0.070770 default_wake_function 0.069429 xfrm4_policy_check 0.069091 tcp_parse_md5sig_option 0.068287 tcp_v4_md5_do_lookup 0.068059 tcp_v4_tw_remember_stamp 0.067344 tcp_ca_event 0.067125 tcp_ca_event 0.065457 place_entity 0.065318 write_seqlock 0.065089 device_not_available 0.065069 test_ti_thread_flag 0.063878 tcp_set_skb_tso_segs 0.063550 selinux_netlbl_inode_permission 0.063391 sock_wfree 0.063311 prepare_to_wait 0.058872 pid_vnr 0.058803 __cycles_2_ns 0.057631 ip_local_out 0.057333 tcp_ack_saw_tstamp 0.056896 copy_to_user 0.056628 set_bit 0.055913 free_pages_check 0.054969 tcp_rcv_rtt_measure_ts 0.053797 init_rootdomain 0.053708 selinux_socket_recvmsg 0.053698 pid_nr_ns 0.053629 sk_eat_skb 0.052814 _local_bh_enable 0.052645 nf_hook_thresh 0.052516 sched_info_queued 0.052457 enqueue_task 0.052228 sk_filter 0.052159 __cpu_clear 0.051980 local_bh_enable_ip 0.050292 update_rq_clock 0.048981 task_tgid_vnr 0.048881 copy_from_user 0.048782 tcp_parse_options 0.048484 lock_sock 0.047779 net_timestamp 0.047044 open_softirq 0.046955 tcp_win_from_space 0.045981 __skb_dequeue 0.043846 getboottime 0.043777 account_group_exec_runtime 0.043519 can_checksum_protocol 0.043469 set_user_nice 0.042784 skb_fill_page_desc 0.042247 security_socket_sendmsg 0.041989 read_profile 0.041930 tcp_validate_incoming 0.041612 check_preempt_curr 0.041413 skb_pull 0.041026 generic_smp_call_function_interrupt 0.041016 calc_delta_fair 0.040936 clear_buddies 0.040768 tcp_data_queue 0.040698 page_count 0.039695 lock_sock 0.039099 skb_headroom 0.038851 system_call_fastpath 0.038622 zone_statistics 0.037500 tcp_sack_extend 0.037381 __kmalloc_node 0.036587 first_zones_zonelist 0.036497 mntput 0.036179 pick_next_task 0.035991 kmap 0.035911 sock_put 0.035613 deactivate_task 0.035027 __nr_to_section 0.033985 page_zone 0.033190 native_load_tls 0.032882 netif_tx_queue_stopped 0.032713 __skb_insert 0.032187 sock_flag 0.031988 check_kill_permission 0.031790 policy_nodemask 0.031621 detach_timer 0.030558 inet_csk_clear_xmit_timer 0.030469 task_rq_unlock 0.029883 tcp_nagle_test 0.029744 tracesys 0.028383 virt_to_slab 0.028115 tcp_v4_check 0.028046 __cpu_set 0.027658 page_get_cache 0.027063 tcp_store_ts_recent 0.027053 __skb_pull 0.026953 gfp_zone 0.026586 sock_rcvlowat 0.026576 csum_partial 0.026397 init_waitqueue_head 0.026109 finish_wait 0.026040 kill_pid_info 0.025404 tcp_full_space 0.024888 __skb_queue_before 0.024550 dst_confirm 0.022603 inet_ehash_bucket 0.021888 activate_task 0.021650 tcp_rto_min 0.021283 d_callback 0.020965 signal_pending 0.020925 avc_node_free 0.020915 empty_bucket 0.020746 group_send_sig_info 0.020657 skb_reset_transport_header 0.020061 sock_put 0.019992 signal_pending_state 0.019684 tcp_sync_mss 0.019346 skb_network_offset 0.019276 skb_split 0.018988 tcp_adjust_fackets_out 0.018204 tcp_fast_path_check 0.017727 __skb_unlink 0.017687 napi_disable_pending 0.017678 sg_set_page 0.017022 get_pageblock_bitmap 0.016972 tcp_cong_avoid 0.016962 pid_task 0.016754 skb_set_tail_pointer 0.016039 selinux_ipv4_postroute 0.015930 idle_cpu 0.015632 skb_reset_network_header 0.015552 __count_vm_events 0.015483 source_load 0.014867 __skb_unlink 0.014738 skb_reset_transport_header 0.014599 set_bit 0.014241 audit_zero_context 0.014231 zone_page_state 0.014152 clear_bit 0.013874 PageSlab 0.013546 __memset 0.013238 get_pageblock_migratetype 0.012623 __rb_rotate_right 0.012543 kmem_find_general_cachep 0.012414 __kprobes_text_start 0.012344 security_sock_rcv_skb 0.012344 node_zonelist 0.012335 dnotify_parent 0.012096 skb_headroom 0.011778 tcp_push_one 0.011540 mnt_want_write 0.011143 kmalloc 0.011073 retint_swapgs 0.010954 __rb_rotate_left 0.010805 check_pgd_range 0.010785 tcp_mss_split_point 0.010755 migrate_timer_list 0.010338 __send_IPI_dest_field 0.010229 reschedule_interrupt 0.010179 sock_flag 0.009882 smp_call_function_mask 0.009673 test_tsk_need_resched 0.009564 tcp_urg 0.009504 generic_file_aio_read 0.009176 PageReserved 0.009147 net_invalid_timestamp 0.009087 __node_set 0.008749 do_tcp_setsockopt 0.008730 set_tsk_thread_flag 0.008720 tcp_enter_loss 0.008422 sock_error 0.008362 target_load 0.008302 crypto_hash_update 0.008104 PageReadahead 0.008044 tcp_poll 0.007915 tcp_checksum_complete 0.007329 tcp_snd_test 0.007309 selinux_file_permission 0.007290 sel_netif_destroy 0.007220 put_pages_list 0.006992 dst_output 0.006743 prepare_to_copy 0.006694 tcp_init_cwnd 0.006555 clear_bit 0.006535 set_bit 0.006425 normal_prio 0.006366 msleep 0.006346 error_sti 0.006336 tcp_rcv_rtt_update 0.006167 tcp_send_ack 0.005989 tcp_init_nondata_skb 0.005720 kfree_skb 0.005502 call_function_interrupt 0.005413 __count_vm_event 0.005403 __skb_checksum_complete_head 0.005363 page_cache_get_speculative 0.005323 dev_kfree_skb_irq 0.005174 skb_store_bits 0.004956 cpu_avg_load_per_task 0.004916 dev_cpu_callback 0.004807 __kmem_cache_destroy 0.004777 tcp_init_metrics 0.004777 io_schedule 0.004777 find_get_page 0.004707 eth_header_parse 0.004688 cap_task_kill 0.004678 error_exit 0.004668 rb_prev 0.004658 tso_fragment 0.004648 mmdrop 0.004628 skb_reset_tail_pointer 0.004598 apic_timer_interrupt 0.004588 clear_bit 0.004519 tcp_simple_retransmit 0.004449 get_max_files 0.004370 sk_stop_timer 0.004340 tcp_reset 0.004251 netlbl_cache_add 0.004201 tcp_add_reno_sack 0.004151 __pskb_trim_head 0.004102 __profile_flip_buffers 0.004092 sk_common_release 0.004052 audit_copy_inode 0.003953 eth_change_mtu 0.003943 vfs_read 0.003923 run_timer_softirq 0.003843 mnt_drop_write 0.003814 clear_page_c 0.003804 do_sync_read 0.003744 unset_migratetype_isolate 0.003714 sk_stream_moderate_sndbuf 0.003545 tcp_try_rmem_schedule 0.003476 native_apic_mem_write 0.003466 sys_read 0.003446 skb_checksum 0.003436 timer_set_base 0.003426 security_task_kill 0.003416 __flow_cache_shrink 0.003406 __skb_checksum_complete 0.003277 alloc_skb 0.003267 physflat_send_IPI_mask 0.003218 skb_gso_ok 0.003178 constant_test_bit 0.003168 find_next_bit 0.003158 selinux_netlbl_skbuff_getsid 0.003118 constant_test_bit 0.003099 pull_task 0.003079 hrtimer_run_queues 0.003049 free_hot_page 0.003009 scheduler_tick 0.002900 set_32bit_tls 0.002890 tcp_acceptable_seq 0.002811 rw_verify_area 0.002751 radix_tree_lookup_slot 0.002731 zero_user_segment 0.002731 sock_common_setsockopt 0.002612 __load_balance_iterator 0.002473 run_posix_cpu_timers 0.002264 task_utime 0.002254 switched_to_fair 0.002185 fsnotify_access 0.002145 __rmqueue_smallest 0.002125 __schedule_bug 0.002095 __task_rq_lock 0.002086 tcp_may_update_window 0.002076 restore_args 0.002066 hrtimer_run_pending 0.002056 generic_segment_checks 0.002026 getnstimeofday 0.002006 idle_task 0.001976 touch_atime 0.001956 __wake_up_locked 0.001927 sk_mem_charge 0.001877 smp_apic_timer_interrupt 0.001827 native_smp_send_reschedule 0.001798 __tcp_fast_path_on 0.001788 file_read_actor 0.001768 _cond_resched 0.001738 avc_policy_seqno 0.001718 tcp_ack_snd_check 0.001629 ip_send_check 0.001619 account_system_time 0.001579 __xapic_wait_icr_idle 0.001579 get_stats 0.001539 tcp_set_state 0.001539 bictcp_state 0.001529 tcp_fast_path_on 0.001519 file_accessed 0.001480 get_seconds 0.001450 kernel_math_error 0.001410 ktime_set 0.001331 kmap_atomic 0.001281 printk_tick 0.001281 __next_cpu_nr 0.001271 account_group_system_time 0.001261 __mod_zone_page_state 0.001222 weighted_cpuload 0.001192 security_file_permission 0.001162 ack_APIC_irq 0.001152 __free_one_page 0.001142 rcu_pending 0.001142 drain_array 0.001122 sched_clock_tick 0.001122 csum_fold 0.001102 ret_from_intr 0.001083 retint_careful 0.001073 need_resched 0.001073 calc_delta_mine 0.001043 tcp_v4_md5_do_del 0.001043 PageActive 0.001033 mark_page_accessed 0.001033 ktime_get_ts 0.001023 tcp_insert_write_queue_after 0.001013 tcp_delack_timer 0.001013 task_tick_fair 0.000973 delay_tsc 0.000963 nv_nic_irq_optimized 0.000904 tick_periodic 0.000894 skb_reserve 0.000884 cache_reap 0.000874 timespec_trunc 0.000864 skb_header_release 0.000854 zone_page_state_add 0.000844 update_process_times 0.000834 sk_rmem_schedule 0.000824 find_busiest_group 0.000804 current_fs_time 0.000785 tick_handle_periodic 0.000785 __sk_mem_schedule 0.000785 irq_enter 0.000755 use_cpu_writer_for_mount 0.000755 tcp_ratehalving_spur_to_response 0.000745 update_wall_time 0.000745 tcp_sendpage 0.000745 __alloc_pages_nodemask 0.000725 ktime_get 0.000725 irq_exit 0.000705 inotify_inode_queue_event 0.000665 set_pageblock_flags_group 0.000646 inotify_dentry_parent_queue_event 0.000626 ack_APIC_irq 0.000606 write_profile 0.000566 set_normalized_timespec 0.000566 raise_softirq 0.000526 task_cputime_zero 0.000516 smp_reschedule_interrupt 0.000516 __skb_insert 0.000497 page_fault 0.000497 __copy_user_nocache 0.000487 run_local_timers 0.000487 read_tsc 0.000487 nf_unregister_hook 0.000477 __rcu_pending 0.000477 jiffies_to_usecs 0.000457 timespec_to_ktime 0.000437 __skb_trim 0.000427 __call_rcu 0.000417 free_pages_bulk 0.000407 smp_call_function_interrupt 0.000397 set_irq_regs 0.000397 radix_tree_deref_slot 0.000397 expand 0.000387 handle_mm_fault 0.000387 handle_IRQ_event 0.000387 fput_light 0.000377 refresh_cpu_vm_stats 0.000377 n_tty_write 0.000367 get_page 0.000358 run_rebalance_domains 0.000358 get_cpu_mask 0.000348 task_hot 0.000348 __skb_queue_after 0.000348 retint_check 0.000348 do_select 0.000338 PageUptodate 0.000338 copy_page_c 0.000328 cond_resched 0.000318 unmap_vmas 0.000318 sk_mem_reclaim 0.000318 rmqueue_bulk 0.000318 reciprocal_value 0.000318 irq_return 0.000308 rb_first 0.000308 alloc_skb 0.000308 account_process_tick 0.000298 net_enable_timestamp 0.000298 clocksource_read 0.000298 account_system_time_scaled 0.000288 sched_slice 0.000278 ip_compute_csum 0.000278 constant_test_bit 0.000278 constant_test_bit 0.000268 set_curr_task_fair 0.000268 note_interrupt 0.000268 exit_idle 0.000258 native_apic_mem_write 0.000258 exit_intr 0.000248 PageReferenced 0.000238 usb_hcd_irq 0.000238 __mnt_is_readonly 0.000238 constant_test_bit 0.000218 IRQ0xba_interrupt 0.000218 handle_fasteoi_irq 0.000209 raise_softirq_irqoff 0.000209 __find_get_block 0.000199 tcp_current_ssthresh 0.000199 n_tty_receive_buf 0.000189 wake_up_page 0.000189 vgacon_save_screen 0.000189 free_block 0.000189 constant_test_bit 0.000179 pagefault_disable 0.000169 clocksource_get_next 0.000169 __bitmap_weight 0.000159 tty_ldisc_deref 0.000159 tcp_write_timer 0.000159 kmem_cache_alloc 0.000159 free_alien_cache 0.000159 ext3_mark_iloc_dirty 0.000159 constant_test_bit 0.000159 __bitmap_equal 0.000149 transfer_objects 0.000149 __rcu_process_callbacks 0.000149 page_waitqueue 0.000149 constant_test_bit 0.000139 __rmqueue 0.000139 release_pages 0.000139 constant_test_bit 0.000129 __tcp_checksum_complete 0.000129 run_workqueue 0.000129 poll_freewait 0.000129 n_tty_read 0.000129 iommu_area_free 0.000129 generic_file_llseek 0.000129 __cpus_setall 0.000129 cond_resched_softirq 0.000129 avc_node_populate 0.000129 add_to_page_cache_lru 0.000129 account_user_time 0.000119 wait_consider_task 0.000119 sys_select 0.000119 round_jiffies_common 0.000119 nv_start_xmit_optimized 0.000119 core_sys_select 0.000109 tcp_tso_segment 0.000109 sigprocmask 0.000109 proc_reg_read 0.000109 path_to_nameidata 0.000109 PageBuddy 0.000109 ohci_irq 0.000109 nv_tx_done_optimized 0.000109 nv_msi_workaround 0.000109 IRQ0xc2_interrupt 0.000109 __ext3_get_inode_loc 0.000109 account_group_user_time 0.000099 __wake_up_sync 0.000099 __up_read 0.000099 update_vsyscall 0.000099 memmove 0.000099 kmalloc 0.000099 ext3_get_blocks_handle 0.000099 do_device_not_available 0.000099 constant_test_bit 0.000089 tcp_incr_quickack 0.000089 smp_send_reschedule 0.000089 remove_from_page_cache 0.000089 rcu_process_callbacks 0.000089 prepare_to_wait_exclusive 0.000089 pde_users_dec 0.000089 find_first_bit 0.000089 constant_test_bit 0.000089 common_interrupt 0.000089 add_wait_queue 0.000079 task_gtime 0.000079 sys_lseek 0.000079 start_this_handle 0.000079 schedule_hrtimeout_range 0.000079 __sched_fork 0.000079 journal_put_journal_head 0.000079 find_first_zero_bit 0.000079 do_syslog 0.000079 do_sync_write 0.000079 constant_test_bit 0.000079 ack_apic_level 0.000070 write_seqlock 0.000070 slab_get_obj 0.000070 remove_wait_queue 0.000070 pty_chars_in_buffer 0.000070 ____pagevec_lru_add 0.000070 lock_hrtimer_base 0.000070 kstat_incr_irqs_this_cpu 0.000070 journal_dirty_data 0.000070 journal_add_journal_head 0.000070 find_lock_page 0.000070 copy_from_read_buf 0.000070 bit_waitqueue 0.000070 alloc_page_vma 0.000060 vfs_write 0.000060 tty_write 0.000060 __strnlen_user 0.000060 sk_mem_uncharge 0.000060 rt_worker_func 0.000060 radix_tree_preload 0.000060 poll_select_copy_remaining 0.000060 pagefault_enable 0.000060 __mark_inode_dirty 0.000060 lru_add_drain_all 0.000060 lock_page 0.000060 list_replace_init 0.000060 journal_stop 0.000060 iowrite8 0.000060 hrtimer_forward 0.000060 gart_unmap_single 0.000060 find_vma 0.000060 __down_read_trylock 0.000060 do_page_fault 0.000060 do_IRQ 0.000060 create_empty_buffers 0.000060 constant_test_bit 0.000060 constant_test_bit 0.000060 alloc_iommu 0.000060 add_to_page_cache_locked 0.000050 zero_fd_set 0.000050 vsnprintf 0.000050 unlock_page 0.000050 tty_read 0.000050 tty_poll 0.000050 sock_poll 0.000050 sock_def_error_report 0.000050 set_wq_data 0.000050 rcu_check_callbacks 0.000050 radix_tree_node_rcu_free 0.000050 pipe_poll 0.000050 opost 0.000050 n_tty_chars_in_buffer 0.000050 __next_cpu 0.000050 mutex_trylock 0.000050 msecs_to_jiffies 0.000050 mempool_alloc_slab 0.000050 load_elf_binary 0.000050 __link_path_walk 0.000050 __journal_remove_journal_head 0.000050 journal_commit_transaction 0.000050 journal_cancel_revoke 0.000050 irq_complete_move 0.000050 irq_cfg 0.000050 fsnotify_modify 0.000050 __first_cpu 0.000050 file_update_time 0.000050 filemap_fault 0.000050 ext3_new_blocks 0.000050 ext3_mark_inode_dirty 0.000050 do_wp_page 0.000050 __do_fault 0.000050 buffer_dirty 0.000050 anon_vma_prepare 0.000040 yield 0.000040 wq_per_cpu 0.000040 walk_page_buffers 0.000040 __wake_up_bit 0.000040 vma_adjust 0.000040 tty_put_char 0.000040 tty_paranoia_check 0.000040 tcp_current_ssthresh 0.000040 sys_write 0.000040 sys_rt_sigprocmask 0.000040 sock_no_bind 0.000040 show_stat 0.000040 SetPageSwapBacked 0.000040 set_irq_regs 0.000040 set_buffer_write_io_error 0.000040 recalc_sigpending 0.000040 radix_tree_delete 0.000040 queue_delayed_work_on 0.000040 pty_write 0.000040 __pollwait 0.000040 physflat_send_IPI_allbutself 0.000040 page_zone 0.000040 page_remove_rmap 0.000040 page_is_file_cache 0.000040 page_evictable 0.000040 nv_get_empty_tx_slots 0.000040 n_tty_poll 0.000040 next_zone 0.000040 next_online_pgdat 0.000040 need_resched 0.000040 mutex_unlock 0.000040 mpol_needs_cond_ref 0.000040 __lookup 0.000040 journal_invalidatepage 0.000040 journal_dirty_metadata 0.000040 ioread8 0.000040 input_available_p 0.000040 inet_csk_reset_xmit_timer 0.000040 get_fd_set 0.000040 generic_write_checks 0.000040 free_poll_entry 0.000040 fput 0.000040 __ext3_journal_stop 0.000040 ext3_get_group_desc 0.000040 ext3_get_block 0.000040 do_mpage_readpage 0.000040 __d_lookup 0.000040 del_page_from_lru 0.000040 __dec_zone_state 0.000040 copy_user_generic 0.000040 __bitmap_and 0.000040 add_page_to_lru_list 0.000040 account_user_time_scaled 0.000040 account_steal_time 0.000030 worker_thread 0.000030 wake_up_bit 0.000030 vmstat_update 0.000030 vm_normal_page 0.000030 tty_write_unlock 0.000030 tty_write_lock 0.000030 tty_wakeup 0.000030 tty_ldisc_try 0.000030 tty_ioctl 0.000030 tag_get 0.000030 sys_pread64 0.000030 submit_bh 0.000030 stop_this_cpu 0.000030 sock_aio_write 0.000030 sk_mem_reclaim 0.000030 sk_backlog_rcv 0.000030 show_interrupts 0.000030 sg_next 0.000030 seq_printf 0.000030 send_remote_softirq 0.000030 remove_vma 0.000030 reg_delay 0.000030 radix_tree_lookup 0.000030 radix_tree_insert 0.000030 proc_lookup_de 0.000030 pipe_write 0.000030 __percpu_counter_add 0.000030 pci_map_single 0.000030 nv_napi_poll 0.000030 __next_node 0.000030 native_send_call_func_ipi 0.000030 mpage_readpages 0.000030 mix_pool_bytes_extract 0.000030 mii_rw 0.000030 mempool_alloc 0.000030 __make_request 0.000030 jbd_lock_bh_state 0.000030 iov_iter_copy_from_user_atomic 0.000030 insert_work 0.000030 hrtimer_try_to_cancel 0.000030 get_dma_ops 0.000030 __generic_file_aio_write_nolock 0.000030 gart_map_sg 0.000030 __fput 0.000030 fixup_irqs 0.000030 __find_get_block_slow 0.000030 filp_close 0.000030 ext3_get_branch 0.000030 ext3_dirty_inode 0.000030 ext3_block_to_path 0.000030 do_get_write_access 0.000030 delayed_work_timer_fn 0.000030 csum_block_add 0.000030 copy_process 0.000030 copy_page_range 0.000030 constant_test_bit 0.000030 constant_test_bit 0.000030 check_irqs_on 0.000030 call_rcu 0.000030 __brelse 0.000030 _atomic_dec_and_lock 0.000020 __xchg 0.000020 vm_stat_account 0.000020 vma_prio_tree_remove 0.000020 tty_mode_ioctl 0.000020 tty_audit_add_data 0.000020 try_to_free_buffers 0.000020 truncate_inode_pages_range 0.000020 tcp_slow_start 0.000020 task_curr 0.000020 sys_setpgid 0.000020 sys_rt_sigreturn 0.000020 sys_getppid 0.000020 strncpy_from_user 0.000020 sock_put 0.000020 smp_call_function 0.000020 __sk_mem_reclaim 0.000020 signal_wake_up 0.000020 signal_pending 0.000020 set_termios 0.000020 SetPageUptodate 0.000020 SetPageLRU 0.000020 set_fd_set 0.000020 set_bit 0.000020 __send_IPI_shortcut 0.000020 security_inode_need_killpriv 0.000020 scsi_request_fn 0.000020 sb_bread 0.000020 restore_i387_xstate 0.000020 __qdisc_run 0.000020 pud_alloc 0.000020 pmd_alloc 0.000020 pfn_pte 0.000020 pfifo_fast_enqueue 0.000020 pfifo_fast_dequeue 0.000020 pci_map_page 0.000020 path_get 0.000020 __pagevec_free 0.000020 pagevec_add 0.000020 PageUnevictable 0.000020 page_mapping 0.000020 nv_get_hw_stats 0.000020 number 0.000020 normalize_rt_tasks 0.000020 __netif_tx_lock 0.000020 mk_pid 0.000020 memscan 0.000020 memcpy_c 0.000020 __lru_cache_add 0.000020 __lookup_mnt 0.000020 load_balance_rt 0.000020 kthread_should_stop 0.000020 journal_start 0.000020 journal_remove_journal_head 0.000020 __journal_file_buffer 0.000020 jbd_unlock_bh_journal_head 0.000020 itimer_get_remtime 0.000020 irq_to_desc 0.000020 iowrite32 0.000020 inotify_remove_watch_locked 0.000020 inode_permission 0.000020 inode_has_perm 0.000020 init_timer 0.000020 goal_in_my_reservation 0.000020 get_vma_policy 0.000020 __get_free_pages 0.000020 generic_sync_sb_inodes 0.000020 gart_map_single 0.000020 freezing 0.000020 free_pgtables 0.000020 free_pages_and_swap_cache 0.000020 free_buffer_head 0.000020 __follow_mount 0.000020 flush_tlb_page 0.000020 find_busiest_queue 0.000020 file_has_perm 0.000020 ext3_try_to_allocate 0.000020 ext3_journal_start 0.000020 __ext3_journal_dirty_metadata 0.000020 ext3_file_write 0.000020 enqueue_hrtimer 0.000020 dup_mm 0.000020 do_wait 0.000020 do_vfs_ioctl 0.000020 do_path_lookup 0.000020 do_munmap 0.000020 do_machine_check 0.000020 do_lookup 0.000020 do_follow_link 0.000020 dma_unmap_single 0.000020 __dec_zone_page_state 0.000020 count_vm_event 0.000020 constant_test_bit 0.000020 constant_test_bit 0.000020 compound_head 0.000020 clear_buffer_jbddirty 0.000020 clear_buffer_delay 0.000020 claim_block 0.000020 cascade 0.000020 cancel_dirty_page 0.000020 cache_grow 0.000020 brelse 0.000020 __block_prepare_write 0.000020 __blocking_notifier_call_chain 0.000020 blk_rq_map_sg 0.000020 __bitmap_empty 0.000020 __bitmap_andnot 0.000020 anon_vma_unlink 0.000010 zone_page_state 0.000010 zero_user_segments 0.000010 __xchg 0.000010 __vma_link_rb 0.000010 vma_link 0.000010 vfs_llseek 0.000010 __up_write 0.000010 update_xtime_cache 0.000010 unmap_underlying_metadata 0.000010 unmap_region 0.000010 unix_poll 0.000010 tty_write_room 0.000010 tty_unthrottle 0.000010 tty_ldisc_ref_wait 0.000010 tty_ldisc_ref 0.000010 tty_fasync 0.000010 tty_check_change 0.000010 tty_chars_in_buffer 0.000010 tty_audit_fork 0.000010 truncate_complete_page 0.000010 test_tsk_thread_flag 0.000010 taskstats_exit 0.000010 sys_writev 0.000010 sys_readahead 0.000010 sys_poll 0.000010 sys_newstat 0.000010 sys_nanosleep 0.000010 sys_ioctl 0.000010 syscall_trace_leave 0.000010 sync_supers 0.000010 stub_execve 0.000010 split_page 0.000010 sock_kfree_s 0.000010 __sleep_on_page_lock 0.000010 skip_atoi 0.000010 signal_pending 0.000010 signal_pending 0.000010 sg_init_table 0.000010 set_task_cpu 0.000010 __set_page_dirty 0.000010 SetPageActive 0.000010 set_bit 0.000010 seq_puts 0.000010 selinux_task_setpgid 0.000010 selinux_secctx_to_secid 0.000010 selinux_sb_show_options 0.000010 selinux_inode_permission 0.000010 selinux_inode_need_killpriv 0.000010 selinux_inode_free_security 0.000010 selinux_inode_alloc_security 0.000010 selinux_d_instantiate 0.000010 security_vm_enough_memory 0.000010 second_overflow 0.000010 scsi_run_queue 0.000010 __scsi_put_command 0.000010 scsi_init_sgtable 0.000010 scsi_end_request 0.000010 schedule_tail 0.000010 schedule_delayed_work 0.000010 sb_any_quota_enabled 0.000010 rt_hash 0.000010 round_jiffies_relative 0.000010 remove_hrtimer 0.000010 __remove_hrtimer 0.000010 __remove_from_page_cache 0.000010 rcu_bh_qsctr_inc 0.000010 radix_tree_tag_clear 0.000010 radix_tree_gang_lookup_tag_slot 0.000010 radix_tree_gang_lookup_slot 0.000010 queue_delayed_work 0.000010 qdisc_run 0.000010 put_tty_queue_nolock 0.000010 put_io_context 0.000010 pty_write_room 0.000010 pty_open 0.000010 ptep_set_access_flags 0.000010 profile_munmap 0.000010 proc_pident_lookup 0.000010 proc_get_inode 0.000010 prio_tree_replace 0.000010 prio_tree_remove 0.000010 prio_tree_insert 0.000010 pmd_none_or_clear_bad 0.000010 pipe_release 0.000010 pipe_read 0.000010 pid_revalidate 0.000010 pgd_alloc 0.000010 pci_unmap_single 0.000010 pci_read_config_dword 0.000010 pci_conf1_write 0.000010 pci_bus_read_config_dword 0.000010 path_walk 0.000010 page_zone 0.000010 PageSwapCache 0.000010 PageSwapCache 0.000010 PageSwapCache 0.000010 __page_set_anon_rmap 0.000010 PagePrivate 0.000010 PagePrivate 0.000010 PagePrivate 0.000010 page_add_file_rmap 0.000010 on_each_cpu 0.000010 nv_do_interrupt 0.000010 net_tx_action 0.000010 netif_start_queue 0.000010 netif_carrier_ok 0.000010 need_resched 0.000010 need_iommu 0.000010 native_pte_clear 0.000010 native_io_delay 0.000010 mutex_lock 0.000010 mprotect_fixup 0.000010 mod_zone_page_state 0.000010 mntput_no_expire 0.000010 mm_init 0.000010 mmap_region 0.000010 mempool_free 0.000010 memcmp 0.000010 mcheck_check_cpu 0.000010 may_open 0.000010 __lookup_tag 0.000010 locks_remove_posix 0.000010 locks_remove_flock 0.000010 lock_buffer 0.000010 load_elf_binary 0.000010 load_balance_fair 0.000010 ll_back_merge_fn 0.000010 kzalloc 0.000010 ktime_add_safe 0.000010 kill_fasync 0.000010 __journal_temp_unlink_buffer 0.000010 journal_switch_revoke_table 0.000010 __journal_remove_checkpoint 0.000010 journal_get_write_access 0.000010 journal_get_undo_access 0.000010 journal_get_descriptor_buffer 0.000010 journal_bmap 0.000010 jbd_unlock_bh_state 0.000010 jbd_unlock_bh_state 0.000010 IRQ0xd2_interrupt 0.000010 ip_append_data 0.000010 iov_iter_advance 0.000010 iov_fault_in_pages_read 0.000010 iommu_area_alloc 0.000010 inode_sub_bytes 0.000010 inode_doinit_with_dentry 0.000010 inode_add_bytes 0.000010 __inc_zone_page_state 0.000010 inc_zone_page_state 0.000010 hweight_long 0.000010 hweight64 0.000010 hrtimer_wakeup 0.000010 hrtimer_init 0.000010 hash_64 0.000010 half_md4_transform 0.000010 __grab_cache_page 0.000010 get_user_pages 0.000010 get_signal_to_deliver 0.000010 get_random_int 0.000010 getname 0.000010 get_empty_filp 0.000010 __getblk 0.000010 generic_permission 0.000010 generic_make_request 0.000010 generic_fillattr 0.000010 generic_file_open 0.000010 generic_file_llseek_unlocked 0.000010 generic_file_buffered_write 0.000010 generic_file_aio_write 0.000010 generic_cont_expand_simple 0.000010 generic_block_bmap 0.000010 freezing 0.000010 free_swap_cache 0.000010 free_pid 0.000010 free_pgd_range 0.000010 free_pages 0.000010 flush_old_exec 0.000010 first_online_pgdat 0.000010 find_vma_prepare 0.000010 find_task_by_pid_type_ns 0.000010 find_next_zero_bit 0.000010 find_inode_fast 0.000010 file_remove_suid 0.000010 file_mask_to_av 0.000010 file_free_rcu 0.000010 __FD_CLR 0.000010 ext3_write_begin 0.000010 ext3_try_to_allocate_with_rsv 0.000010 ext3_ordered_write_end 0.000010 ext3_journalled_set_page_dirty 0.000010 ext3_invalidatepage 0.000010 ext3_iget_acl 0.000010 ext3_get_inode_flags 0.000010 ext3_free_data 0.000010 ext3_discard_reservation 0.000010 exit_thread 0.000010 exit_task_namespaces 0.000010 exit_sem 0.000010 end_that_request_last 0.000010 end_buffer_write_sync 0.000010 end_buffer_async_write 0.000010 elv_rb_del 0.000010 elv_queue_empty 0.000010 elv_merged_request 0.000010 elv_completed_request 0.000010 elf_map 0.000010 echo_char 0.000010 e1000_watchdog 0.000010 e1000_read_phy_reg 0.000010 __drain_alien_cache 0.000010 __d_path 0.000010 __down_write_nested 0.000010 __down_write 0.000010 double_rq_lock 0.000010 do_timer 0.000010 do_sys_open 0.000010 do_sigaltstack 0.000010 do_sigaction 0.000010 do_setitimer 0.000010 do_pipe_flags 0.000010 __do_page_cache_readahead 0.000010 do_notify_parent 0.000010 do_filp_open 0.000010 do_exit 0.000010 dnotify_flush 0.000010 d_kill 0.000010 destroy_inode 0.000010 dequeue_signal 0.000010 de_put 0.000010 delayacct_end 0.000010 create_write_pipe 0.000010 create_workqueue_thread 0.000010 __cpus_equal 0.000010 cpu_quiet 0.000010 __cpu_clear 0.000010 __cpu_clear 0.000010 count 0.000010 copy_thread 0.000010 copy_namespaces 0.000010 constant_test_bit 0.000010 constant_test_bit 0.000010 constant_test_bit 0.000010 constant_test_bit 0.000010 constant_test_bit 0.000010 __cond_resched 0.000010 clocksource_forward_now 0.000010 __clear_user 0.000010 clear_inode 0.000010 clear_buffer_new 0.000010 clear_bit 0.000010 clear_bit 0.000010 check_for_bios_corruption 0.000010 __cfq_slice_expired 0.000010 cfq_set_request 0.000010 cfq_dispatch_requests 0.000010 cfq_completed_request 0.000010 cap_set_effective 0.000010 can_share_swap_page 0.000010 bvec_alloc_bs 0.000010 buffer_uptodate 0.000010 buffer_mapped 0.000010 buffer_locked 0.000010 buffer_jbd 0.000010 buffer_jbd 0.000010 brelse 0.000010 __bread 0.000010 blk_invoke_request_fn 0.000010 __blk_complete_request 0.000010 blk_add_trace_generic 0.000010 blk_add_trace_bio 0.000010 bit_spin_lock 0.000010 bio_put 0.000010 bio_alloc_bioset 0.000010 bdi_read_congested 0.000010 balance_runtime 0.000010 balance_dirty_pages_ratelimited_nr 0.000010 audit_log_task_context 0.000010 ata_sff_qc_prep 0.000010 ata_scsi_queuecmd 0.000010 ata_link_max_devices 0.000010 ata_get_xlat_func 0.000010 arp_process 0.000010 arch_pick_mmap_layout 0.000010 arch_irq_stat_cpu 0.000010 arch_dup_task_struct 0.000010 alloc_pid 0.000010 alloc_fdtable 0.000010 alloc_fd 0.000010 add_mm_rss 0.000010 acct_collect -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/