I started linux-2.4.22-pre5 at 1:00am it stopped at 8:00am, with no error
messages. I did enable sysrq keys here is the information.
Show Memory
Jul 12 10:42:41 server SysRq : Show Memory
Jul 12 10:42:41 server Mem-info:
Jul 12 10:42:41 server Free pages: 507876kB ( 2044kB HighMem)
Jul 12 10:42:41 server Zone:DMA freepages: 13600kB min: 128kB low: 256kB
high: 384kB
Jul 12 10:42:41 server Zone:Normal freepages:492232kB min: 1020kB low:
2040kB high: 3060kB
Jul 12 10:42:41 server Zone:HighMem freepages: 2044kB min: 1020kB low:
2040kB high: 3060kB
Jul 12 10:42:41 server ( Active: 41175, inactive: 52516, free: 126969 )
Jul 12 10:42:41 server 2*4kB 3*8kB 2*16kB 3*32kB 0*64kB 1*128kB 2*256kB
1*512kB 0*1024kB 6*2048kB = 13600kB)
Jul 12 10:42:41 server 35*4kB 62*8kB 25*16kB 58*32kB 102*64kB 50*128kB
13*256kB 4*512kB 2*1024kB 229*2048kB = 492236kB)
Jul 12 10:42:41 server 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB
1*512kB 1*1024kB 0*2048kB = 2044kB)
Jul 12 10:42:41 server Swap cache: add 0, delete 0, find 0/0, race 0+0
Jul 12 10:42:41 server Free swap: 265060kB
Jul 12 10:42:41 server 262144 pages of RAM
Jul 12 10:42:41 server 32768 pages of HIGHMEM
Jul 12 10:42:41 server 3726 reserved pages
Jul 12 10:42:41 server 123007 pages shared
Jul 12 10:42:41 server 0 pages swap cached
Jul 12 10:42:41 server 34 pages in page table cache
Jul 12 10:42:41 server Buffer memory: 128776kB
Jul 12 10:42:41 server Cache memory: 130604kB
Jul 12 10:42:41 server CLEAN: 12454 buffers, 49813 kbyte, 52 used
(last=12452), 0 locked, 0 dirty
Jul 12 10:42:41 server LOCKED: 27 buffers, 96 kbyte, 25 used (last=27), 0
locked, 0 dirty
Jul 12 10:42:41 server DIRTY: 12 buffers, 48 kbyte, 12 used (last=12), 0
locked, 8 dirty
Show Regs
Jul 12 10:42:51 server SysRq : Show Regs
Jul 12 10:42:51 server
Jul 12 10:42:51 server Pid: 3216, comm: setiathome
Jul 12 10:42:51 server EIP: 0023:[<0806330d>] CPU: 1 ESP: 002b:bffff1ec
EFLAGS: 00000206 Tainted: P
Jul 12 10:42:51 server EAX: 40ff2040 EBX: 00022138 ECX: 00022158 EDX:
00022168
Jul 12 10:42:51 server ESI: 00022148 EDI: 00022128 EBP: bffff298 DS: 002b
ES: 002b
Jul 12 10:42:51 server CR0: 80050033 CR2: 41072000 CR3: 36d36000 CR4:
000002d0
Show State
Jul 12 10:42:54 server SysRq : Show State
Jul 12 10:42:54 server
Jul 12 10:42:54 server free sibling
Jul 12 10:42:54 server task PC stack pid father child
younger older
Jul 12 10:42:54 server init S C0270798 3072 1 0 5856
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011c858>] [<c01601cc>] [<c011c7e0>]
[<c0160437>] [<c016089d>]
Jul 12 10:42:54 server [<c0159cc6>] [<c0109a3f>]
Jul 12 10:42:54 server keventd S C1C1DF54 5880 2 1
3 (L-TLB)
Jul 12 10:42:54 server Call Trace: [<c01b690d>] [<c012c203>] [<c0131924>]
[<c01317a0>] [<c010788e>]
Jul 12 10:42:54 server [<c01317a0>]
Jul 12 10:42:54 server ksoftirqd_CPU S C1C1BFAC 6044 3 1
4 2 (L-TLB)
Jul 12 10:42:54 server Call Trace: [<c01eb557>] [<c0127a2c>] [<c010788e>]
[<c0127940>]
Jul 12 10:42:54 server ksoftirqd_CPU S F6D34000 6208 4 1
5 3 (L-TLB)
Jul 12 10:42:54 server Call Trace: [<c0127a2c>] [<c010788e>] [<c0127940>]
Jul 12 10:42:54 server kswapd S F7F5A000 6300 5 1
6 4 (L-TLB)
Jul 12 10:42:54 server Call Trace: [<c0124e00>] [<c0141b59>] [<c0141aa0>]
[<c010788e>] [<c0141aa0>]
Jul 12 10:42:54 server bdflush S FFFFFFFF 6276 6 1
7 5 (L-TLB)
Jul 12 10:42:54 server Call Trace: [<c011d330>] [<c015245f>] [<c010788e>]
[<c0152310>]
Jul 12 10:42:54 server kupdated S 001F5ECD 5372 7 1
19 6 (L-TLB)
Jul 12 10:42:54 server Call Trace: [<c0185c6e>] [<c011c858>] [<c011c7e0>]
[<c0152575>] [<c0152480>]
Jul 12 10:42:54 server [<c010788e>] [<c0152480>]
Jul 12 10:42:54 server ahc_dv_0 S F7B67CC4 5556 17 1
22 18 (L-TLB)
Jul 12 10:42:54 server Call Trace: [<c010827e>] [<fa80b373>] [<c0108412>]
[<fa82d5ef>] [<fa827ee0>]
Jul 12 10:42:54 server [<c010788e>] [<fa827ee0>]
Jul 12 10:42:54 server ahc_dv_1 S FFFFFFFF 5984 18 1
17 20 (L-TLB)
Jul 12 10:42:54 server Call Trace: [<c010827e>] [<c011e306>] [<c0108412>]
[<fa82d5ef>] [<fa827ee0>]
Jul 12 10:42:54 server [<c010788e>] [<fa827ee0>]
Jul 12 10:42:54 server scsi_eh_1 S FFFFFFFF 6168 19 1
20 7 (L-TLB)
Jul 12 10:42:54 server Call Trace: [<c010827e>] [<c0108412>] [<fa80f100>]
[<fa808daf>] [<fa80f186>]
Jul 12 10:42:54 server [<c01099f2>] [<c010788e>] [<fa8088d0>]
Jul 12 10:42:54 server scsi_eh_2 S FFFFFFFF 6168 20 1
18 19 (L-TLB)
Jul 12 10:42:54 server Call Trace: [<c010827e>] [<c0108412>] [<fa80f100>]
[<fa808daf>] [<fa80f186>]
Jul 12 10:42:54 server [<c01099f2>] [<c010788e>] [<fa8088d0>]
Jul 12 10:42:54 server khubd S C02B0000 2384 22 1
32 17 (L-TLB)
Jul 12 10:42:54 server Call Trace: [<c0124e00>] [<fa85ba36>] [<c010ef5a>]
[<fa866838>] [<fa866838>]
Jul 12 10:42:54 server [<c010788e>] [<fa85b8e0>]
Jul 12 10:42:54 server kjournald S F6D34000 4484 32 1
51 22 (L-TLB)
Jul 12 10:42:54 server Call Trace: [<c01193cc>] [<c011d330>] [<c018cdc9>]
[<c01099f2>] [<c018cbd0>]
Jul 12 10:42:54 server [<c010788e>] [<c018cbf0>]
Jul 12 10:42:54 server devfsd S F7332000 5144 51 1 5030
74 32 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0109a3f>]
Jul 12 10:42:54 server kjournald S F7538000 5568 74 1
75 51 (L-TLB)
Jul 12 10:42:54 server Call Trace: [<c01219dd>] [<c011d330>] [<c0121805>]
[<c018cdc9>] [<c01099f2>]
Jul 12 10:42:54 server [<c018cbd0>] [<c010788e>] [<c018cbf0>]
Jul 12 10:42:54 server kjournald S C1C34000 4592 75 1
76 74 (L-TLB)
Jul 12 10:42:54 server Call Trace: [<c011d330>] [<c018cdc9>] [<c01099f2>]
[<c018cbd0>] [<c010788e>]
Jul 12 10:42:54 server [<c018cbf0>]
Jul 12 10:42:54 server kjournald S F7404000 4296 76 1
227 75 (L-TLB)
Jul 12 10:42:54 server Call Trace: [<c011d330>] [<c018cdc9>] [<c01099f2>]
[<c018cbd0>] [<c010788e>]
Jul 12 10:42:54 server [<c018cbf0>]
Jul 12 10:42:54 server gpm S 00000282 1360 227 1
245 76 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011c858>] [<c0234f9e>] [<c011c7e0>]
[<c01e299c>] [<c0160437>]
Jul 12 10:42:54 server [<c016089d>] [<c014b628>] [<c0109a3f>]
Jul 12 10:42:54 server syslog-ng R F6D34000 0 245 1
297 227 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011cc0b>] [<c0142c37>] [<c01a9ecd>]
[<c01e299c>] [<c0160144>]
Jul 12 10:42:54 server [<c0160dde>] [<c014c1c7>] [<c01269da>] [<c0109a3f>]
Jul 12 10:42:54 server dhcpd S 00000000 4576 297 1
1933 245 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0142acc>] [<c0142c37>] [<c011c858>]
[<c01e928e>] [<c011c7e0>]
Jul 12 10:42:54 server [<c01e299c>] [<c0160437>] [<c016089d>] [<c0109a3f>]
Jul 12 10:42:54 server zorbcount.pl S F7427560 1360 1933 1 5024
1940 297 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011bab0>] [<c0158ab8>] [<c0158bbb>]
[<c014c10d>] [<c0109a3f>]
Jul 12 10:42:54 server safe_mysqld S F7880000 1360 1940 1 1960
1961 1933 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c012e35d>] [<c0125c94>] [<c012d630>]
[<c0109a3f>]
Jul 12 10:42:54 server mysqld S F7881ECC 2384 1960 1940 1970
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011c8a5>] [<c0234f9e>] [<c01e299c>]
[<c0160437>] [<c016089d>]
Jul 12 10:42:54 server [<c0109a3f>]
Jul 12 10:42:54 server clamd S F7B261D4 5132 1961 1 1962
2029 1940 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c01872ba>] [<c0187b0e>] [<c011c8a5>]
[<c020a158>] [<c01807ee>]
Jul 12 10:42:54 server [<c020a295>] [<c0166a04>] [<c0228520>] [<c01e2269>]
[<c01e333a>] [<c011cc0b>]
Jul 12 10:42:54 server [<c012e33c>] [<c01e3f12>] [<c0107d1d>] [<c0109a3f>]
Jul 12 10:42:54 server clamd Z F7A49F3C 4616 1962 1961
(L-TLB)
Jul 12 10:42:54 server Call Trace: [<c0125383>] [<c011ef3f>] [<c0125a40>]
[<c0125b65>] [<c0109a3f>]
Jul 12 10:42:54 server clamd S F6D34000 4448 1964 1
5856 22759 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c012b556>] [<c011c858>] [<c01193cc>]
[<c011c7e0>] [<c012d630>]
Jul 12 10:42:54 server [<c012b9bb>] [<c0109a3f>]
Jul 12 10:42:54 server mysqld S F6D22000 5060 1970 1960 30425
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<c01601cc>] [<c011c858>] [<c011c7e0>]
[<c0160b7e>] [<c0160d3a>]
Jul 12 10:42:54 server [<c0109a3f>]
Jul 12 10:42:54 server mysqld S 00000000 2388 1971 1970
1972 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c010878b>] [<c0109a3f>]
Jul 12 10:42:54 server mysqld S 00000000 2384 1972 1970
1973 1971 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c010878b>] [<c0109a3f>]
Jul 12 10:42:54 server mysqld S C0138B40 5428 1973 1970
1974 1972 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0138b40>] [<c010878b>] [<c0109a3f>]
Jul 12 10:42:54 server mysqld S 00000000 2388 1974 1970
1975 1973 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c010878b>] [<c0109a3f>]
Jul 12 10:42:54 server mysqld S 00000000 5836 1975 1970
1976 1974 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0160812>] [<c010878b>] [<c0109a3f>]
Jul 12 10:42:54 server mysqld S F6D34000 4632 1976 1970
1977 1975 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c01fd050>] [<c01fce6c>] [<c011c858>]
[<c011c7e0>] [<c0160437>]
Jul 12 10:42:54 server [<c016089d>] [<c012b556>] [<c0109a3f>]
Jul 12 10:42:54 server mysqld S 00000001 4952 1977 1970
1978 1976 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c010878b>] [<c0109a3f>]
Jul 12 10:42:54 server mysqld S 00000286 2384 1978 1970
1979 1977 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c012629b>] [<c01264b2>] [<c010878b>]
[<c0109a3f>]
Jul 12 10:42:54 server mysqld S F6D22000 4980 1979 1970
3089 1978 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011c8a5>] [<c023470a>] [<c0234c5b>]
[<c01e249f>] [<c01e25c6>]
Jul 12 10:42:54 server [<c014c10d>] [<c0109a3f>]
Jul 12 10:42:54 server spamd S 00000000 4880 2029 1
2107 1961 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011c8a5>] [<c012c6d2>] [<c01e299c>]
[<c0160437>] [<c016089d>]
Jul 12 10:42:54 server [<c0109a3f>]
Jul 12 10:42:54 server courierfilter S 00000000 1360 2107 1
2112 2029 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c010878b>] [<c0109a3f>]
Jul 12 10:42:54 server courierlogger S F7245EFC 4880 2112 1
2124 2107 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0134d24>] [<c0158ab8>] [<c0158bbb>]
[<c014c10d>] [<c0109a3f>]
Jul 12 10:42:54 server authdaemond.m S C0270798 1360 2124 1 2132
2150 2112 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011c858>] [<c01601cc>] [<c011c7e0>]
[<c0160437>] [<c016089d>]
Jul 12 10:42:54 server [<c011cc0b>] [<c0109a3f>]
Jul 12 10:42:54 server authdaemond.m S F7269EEC 2384 2125 2124
2126 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011c858>] [<c0234f9e>] [<c011c7e0>]
[<c01e299c>] [<c0160437>]
Jul 12 10:42:54 server [<c016089d>] [<c0109a3f>]
Jul 12 10:42:54 server authdaemond.m S C0270798 4880 2126 2124
2127 2125 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011c858>] [<c0234f9e>] [<c011c7e0>]
[<c01e299c>] [<c0160437>]
Jul 12 10:42:54 server [<c016089d>] [<c0109a3f>]
Jul 12 10:42:54 server authdaemond.m S C0270798 4996 2127 2124
2131 2126 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011c858>] [<c0234f9e>] [<c011c7e0>]
[<c01e299c>] [<c0160437>]
Jul 12 10:42:54 server [<c016089d>] [<c0109a3f>]
Jul 12 10:42:54 server authdaemond.m S C0270798 4788 2131 2124
2132 2127 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011c858>] [<c0234f9e>] [<c011c7e0>]
[<c01e299c>] [<c0160437>]
Jul 12 10:42:54 server [<c016089d>] [<c0109a3f>]
Jul 12 10:42:54 server authdaemond.m S C0270798 4956 2132 2124
2131 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011c858>] [<c0234f9e>] [<c011c7e0>]
[<c01e299c>] [<c0160437>]
Jul 12 10:42:54 server [<c016089d>] [<c0109a3f>]
Jul 12 10:42:54 server courierd S F7232000 4248 2150 1 2588
2170 2124 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S C012DF55 4840 2170 1 5725
2179 2150 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c012df55>] [<c012e35d>] [<c0125c94>]
[<c012d630>] [<c0109a3f>]
Jul 12 10:42:54 server courierlogger S F719E000 1360 2179 1
2194 2170 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011bab0>] [<c0158ab8>] [<c0158bbb>]
[<c014c10d>] [<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S F71A3EB4 4960 2194 1
2205 2179 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011c8a5>] [<c01e299c>] [<c0160437>]
[<c016089d>] [<c0109a3f>]
Jul 12 10:42:54 server courierlogger S F7165EFC 4824 2205 1
2221 2194 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0134d24>] [<c0158ab8>] [<c0158bbb>]
[<c014c10d>] [<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S F7135EB4 5132 2221 1
2231 2205 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011c8a5>] [<c01e299c>] [<c0160437>]
[<c016089d>] [<c0109a3f>]
Jul 12 10:42:54 server courierlogger S F7145EFC 4880 2231 1
2243 2221 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0134d24>] [<c0158ab8>] [<c0158bbb>]
[<c014c10d>] [<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S F7179EB4 2344 2243 1
2252 2231 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011c8a5>] [<c01e299c>] [<c0160437>]
[<c016089d>] [<c0109a3f>]
Jul 12 10:42:54 server courierlogger S F7193EFC 4880 2252 1
2269 2243 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0134d24>] [<c0158ab8>] [<c0158bbb>]
[<c014c10d>] [<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S F712A668 2388 2269 1
2279 2252 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011c8a5>] [<c011bab0>] [<c01e299c>]
[<c0160437>] [<c016089d>]
Jul 12 10:42:54 server [<c0109a3f>]
Jul 12 10:42:54 server courierlogger S F7332000 4880 2279 1
2295 2269 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011cc64>] [<c0158ab8>] [<c0158bbb>]
[<c014c10d>] [<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S F70EC668 4196 2295 1 5054
2305 2279 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011c8a5>] [<c011bab0>] [<c01e299c>]
[<c0160437>] [<c016089d>]
Jul 12 10:42:54 server [<c0109a3f>]
Jul 12 10:42:54 server courierlogger S F6D22000 1360 2305 1
2318 2295 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0158ab8>] [<c0158bbb>] [<c014c10d>]
[<c0109a3f>]
Jul 12 10:42:54 server fcron S F6D22000 1360 2318 1 5178
2324 2305 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011c858>] [<c011c7e0>] [<c012d514>]
[<c012b9bb>] [<c0109a3f>]
Jul 12 10:42:54 server cupsd S F709E005 1360 2324 1 2918
2927 2318 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server foomatic-rip S F7427268 4248 2917 2324 2959
2918 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011bab0>] [<c0158ab8>] [<c0158bbb>]
[<c014c10d>] [<c0109a3f>]
Jul 12 10:42:54 server parallel D 00000001 4880 2918 2324
2917 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0197e36>] [<c0197d0c>] [<c0199f10>]
[<c0159e6f>] [<c0159d0b>]
Jul 12 10:42:54 server [<c015a806>] [<fa91888e>] [<fa918bd0>] [<fa918de1>]
[<c0109a3f>]
Jul 12 10:42:54 server portmap S F703E768 2384 2927 1
2946 2324 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c01e928e>] [<c011c8a5>] [<c01e299c>]
[<c0160a9b>] [<c0160b7e>]
Jul 12 10:42:54 server [<c0160d3a>] [<c0109a3f>]
Jul 12 10:42:54 server chronyd S F7ABE400 4248 2946 1
2968 2927 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<c0164910>] [<c014d837>]
[<fa9185e1>] [<c014b628>]
Jul 12 10:42:54 server [<fa91906f>] [<c0109a3f>]
Jul 12 10:42:54 server foomatic-rip S C012DF55 5544 2959 2917 2961
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<c012df55>] [<c012e35d>] [<c0125c94>]
[<c0109a3f>]
Jul 12 10:42:54 server foomatic-rip S 00000000 5912 2960 2959
2961 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c012be2c>] [<c0158ab8>] [<c0158f7b>]
[<c014c2ed>] [<c0109a3f>]
Jul 12 10:42:54 server sh S F6FE8000 4880 2961 2959 2964
2960 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c012e35d>] [<c0125c94>] [<c012d630>]
[<c0109a3f>]
Jul 12 10:42:54 server sh S F6FCA000 4636 2963 2961 2972
2964 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c012e35d>] [<c0125c94>] [<c012d630>]
[<c0109a3f>]
Jul 12 10:42:54 server pnm2ppa S F6FF6000 4248 2964 2961
2963 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0158ab8>] [<c0158ed2>] [<c014c2ed>]
[<c0109a3f>]
Jul 12 10:42:54 server nmbd S C0270798 4336 2968 1 2970
2997 2946 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011c858>] [<c011c7e0>] [<c0160437>]
[<c016089d>] [<c0109a3f>]
Jul 12 10:42:54 server nmbd S F7427D18 4824 2970 2968
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011bab0>] [<c0158ab8>] [<c0158bbb>]
[<c014c10d>] [<c0109a3f>]
Jul 12 10:42:54 server gs S F6F0A000 4248 2972 2963 3029
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<c012be2c>] [<c0158ab8>] [<c0158f7b>]
[<c014c2ed>] [<c01193cc>]
Jul 12 10:42:54 server [<c0109a3f>]
Jul 12 10:42:54 server smbd S F7A6BA34 4428 2997 1 5730
3111 2968 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011c8a5>] [<c0160437>] [<c016089d>]
[<fa91a238>] [<c0109a3f>]
Jul 12 10:42:54 server sh S F6F0A000 4608 3029 2972 3033
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c012d630>] [<c0109a3f>]
Jul 12 10:42:54 server cat S 00000000 4880 3033 3029
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0108317>] [<c0158ab8>] [<c0158f7b>]
[<c014c2ed>] [<c01193cc>]
Jul 12 10:42:54 server [<c0109a3f>]
Jul 12 10:42:54 server mysqld S F7220000 5412 3089 1970
3631 1979 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011c8a5>] [<c023470a>] [<c0234c5b>]
[<c01e249f>] [<c01e25c6>]
Jul 12 10:42:54 server [<c014c10d>] [<c0109a3f>]
Jul 12 10:42:54 server httpd S F6D22000 4824 3111 1 5848
3178 2997 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011c858>] [<c011c7e0>] [<c0160437>]
[<c016089d>] [<c0125cb3>]
Jul 12 10:42:54 server [<c0109a3f>]
Jul 12 10:42:54 server httpd S F709E005 3968 3137 3111
3138 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0135b21>] [<c0135f12>]
Jul 12 10:42:54 server [<c0109a3f>]
Jul 12 10:42:54 server httpd S 00000004 4412 3138 3111
3141 3137 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0201480>] [<c020009c>] [<c011c8a5>]
[<fa897f07>] [<c020a158>]
Jul 12 10:42:54 server [<c01eb557>] [<c020a295>] [<c0166a04>] [<c0228520>]
[<c01e2269>] [<c01e333a>]
Jul 12 10:42:54 server [<c0164910>] [<c014d837>] [<c01688ba>] [<c0158b2b>]
[<c01e3f12>] [<c0109a3f>]
Jul 12 10:42:54 server httpd S 00000004 4360 3141 3111
3142 3138 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0201480>] [<c020009c>] [<c011c8a5>]
[<c020a158>] [<c01e66a3>]
Jul 12 10:42:54 server [<c020a295>] [<c0166a04>] [<c0228520>] [<c01e2269>]
[<c01e333a>] [<c0164910>]
Jul 12 10:42:54 server [<c014d837>] [<c01688ba>] [<c0158b2b>] [<c01e3f12>]
[<c0109a3f>]
Jul 12 10:42:54 server httpd S F709E005 4488 3142 3111
3143 3141 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server httpd S F709E005 4032 3143 3111
8526 3142 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server miniserv.pl S F6D22000 4616 3178 1
3198 3111 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011c858>] [<c01e928e>] [<c011c7e0>]
[<c01e299c>] [<c0160437>]
Jul 12 10:42:54 server [<c016089d>] [<c0109a3f>]
Jul 12 10:42:54 server miniserv.pl S F6D34000 4960 3198 1
3216 3178 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011c858>] [<c01e928e>] [<c011c7e0>]
[<c01e299c>] [<c0160437>]
Jul 12 10:42:54 server [<c016089d>] [<c0109a3f>]
Jul 12 10:42:54 server setiathome R current 4240 3215 1
3223 3216 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c010b79c>] [<c010e628>]
Jul 12 10:42:54 server setiathome R F7AD8000 1360 3216 1
3215 3198 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c01193cc>] [<c0109aed>]
Jul 12 10:42:54 server xinetd S C0270798 4996 3223 1
3248 3215 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011c8a5>] [<c01601cc>] [<c01e299c>]
[<c0160437>] [<c016089d>]
Jul 12 10:42:54 server [<c0109a3f>]
Jul 12 10:42:54 server sshd S 00000000 4 3248 1 11222
3269 3223 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011c8a5>] [<c011bab0>] [<c01e299c>]
[<c0160437>] [<c016089d>]
Jul 12 10:42:54 server [<c0109a3f>]
Jul 12 10:42:54 server proftpd S F6D22000 0 3269 1
3270 3248 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011c858>] [<c011c7e0>] [<c01e299c>]
[<c0160437>] [<c016089d>]
Jul 12 10:42:54 server [<c0109a3f>]
Jul 12 10:42:54 server agetty S F6FD2005 4616 3270 1
3271 3269 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c012ffe9>] [<c0109a3f>]
Jul 12 10:42:54 server agetty S 00000008 4636 3271 1
3272 3270 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c01b7094>] [<c011c8a5>] [<c01a9d16>]
[<c01a9a57>] [<c01a494e>]
Jul 12 10:42:54 server [<c014c10d>] [<c012ffe9>] [<c0109a3f>]
Jul 12 10:42:54 server agetty S 00000008 16 3272 1
3273 3271 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c01b7094>] [<c011c8a5>] [<c01a9d16>]
[<c01a9a57>] [<c01a494e>]
Jul 12 10:42:54 server [<c014c10d>] [<c012ffe9>] [<c0109a3f>]
Jul 12 10:42:54 server agetty S 00000008 0 3273 1
3274 3272 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c01b7094>] [<c011c8a5>] [<c01a9d16>]
[<c01a9a57>] [<c01a494e>]
Jul 12 10:42:54 server [<c014c10d>] [<c012ffe9>] [<c0109a3f>]
Jul 12 10:42:54 server agetty S 00000008 0 3274 1
3275 3273 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c01b7094>] [<c011c8a5>] [<c01a9d16>]
[<c01a9a57>] [<c01a494e>]
Jul 12 10:42:54 server [<c014c10d>] [<c012ffe9>] [<c0109a3f>]
Jul 12 10:42:54 server agetty S 00000008 4 3275 1
3276 3274 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c01b7094>] [<c011c8a5>] [<c01a9d16>]
[<c01a9a57>] [<c01a494e>]
Jul 12 10:42:54 server [<c014c10d>] [<c012ffe9>] [<c0109a3f>]
Jul 12 10:42:54 server mgetty S F5835EC0 1536 3276 1
3277 3275 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0142acc>] [<c0142c37>] [<c011c858>]
[<c01a9ecd>] [<c011c7e0>]
Jul 12 10:42:54 server [<c01a5c86>] [<c0160437>] [<c016089d>] [<c0109a3f>]
Jul 12 10:42:54 server mgetty S EA507FDC 0 3277 1
22759 3276 (L-TLB)
Jul 12 10:42:54 server Call Trace: [<c01872ba>] [<c011c8a5>] [<c01872ba>]
[<c01aad4f>] [<c01bbcaf>]
Jul 12 10:42:54 server [<c01a56d4>] [<c0140de3>] [<c017e59e>] [<c01a5bb8>]
[<c014d865>] [<c014b54f>]
Jul 12 10:42:54 server [<c0124d04>] [<c0125824>] [<c014c3a7>] [<c0125b65>]
[<c0109a3f>]
Jul 12 10:42:54 server mysqld S F6D34000 0 3631 1970
4326 3089 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011c8a5>] [<c023470a>] [<c0234c5b>]
[<c01e249f>] [<c01e25c6>]
Jul 12 10:42:54 server [<c014c10d>] [<c0109a3f>]
Jul 12 10:42:54 server mysqld S F7234000 0 4326 1970
22814 3631 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011c8a5>] [<c023470a>] [<c0234c5b>]
[<c01e249f>] [<c01e25c6>]
Jul 12 10:42:54 server [<c014c10d>] [<c0109a3f>]
Jul 12 10:42:54 server sshd S F4AD0000 0 11222 3248 11320
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011c8a5>] [<c01a9ecd>] [<c0160437>]
[<c016089d>] [<c0109a3f>]
Jul 12 10:42:54 server bash S 00000010 2100 11320 11222
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011c8a5>] [<c01a9d16>] [<c01a9a57>]
[<c01a494e>] [<c014c10d>]
Jul 12 10:42:54 server [<c01269da>] [<c0109a3f>]
Jul 12 10:42:54 server named S 00000000 0 22759 1
1964 3277 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0142acc>] [<c011c858>] [<c01e928e>]
[<c011c7e0>] [<c01e299c>]
Jul 12 10:42:54 server [<c0160437>] [<c016089d>] [<c0109a3f>]
Jul 12 10:42:54 server mysqld S F7268000 0 22814 1970
30425 4326 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011c8a5>] [<c023470a>] [<c0234c5b>]
[<c01e249f>] [<c01e25c6>]
Jul 12 10:42:54 server [<c014c10d>] [<c0109a3f>]
Jul 12 10:42:54 server mysqld S F7238000 0 30425 1970
22814 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011c8a5>] [<c023470a>] [<c0234c5b>]
[<c01e249f>] [<c01e25c6>]
Jul 12 10:42:54 server [<c014c10d>] [<c0109a3f>]
Jul 12 10:42:54 server httpd S F709E005 2100 8526 3111
16297 3143 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0135b21>] [<c0135f12>]
Jul 12 10:42:54 server [<c0109a3f>]
Jul 12 10:42:54 server httpd S 00000004 0 16297 3111
16325 8526 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0201480>] [<c020009c>] [<c011c8a5>]
[<c0201688>] [<c020a158>]
Jul 12 10:42:54 server [<c020a295>] [<c0166a04>] [<c0228520>] [<c01e2269>]
[<c01e333a>] [<c0164910>]
Jul 12 10:42:54 server [<c014d837>] [<c014b54f>] [<c01e3f12>] [<c0109a3f>]
Jul 12 10:42:54 server httpd S F709E005 0 16325 3111
5309 16297 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0135b21>] [<c0135f12>]
Jul 12 10:42:54 server [<c0109a3f>]
Jul 12 10:42:54 server courierd S C0270798 4608 2588 2150 2593
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011c8a5>] [<c01601cc>] [<c0160437>]
[<c016089d>] [<c0109a3f>]
Jul 12 10:42:54 server courieruucp S E7E5BEFC 0 2589 2588
2590 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0134d24>] [<c0158ab8>] [<c0158bbb>]
[<c014c10d>] [<c0109a3f>]
Jul 12 10:42:54 server courierlocal S 0000001C 8 2590 2588
2591 2589 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0158ab8>] [<c0158bbb>] [<c014c10d>]
[<c0109a3f>]
Jul 12 10:42:54 server courierfax S E9DAFEFC 0 2591 2588
2592 2590 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0134d24>] [<c0158ab8>] [<c0158bbb>]
[<c014c10d>] [<c0109a3f>]
Jul 12 10:42:54 server courieresmtp S C0270798 0 2592 2588 5034
2593 2591 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c011c8a5>] [<c01601cc>] [<c0160437>]
[<c016089d>] [<c0109a3f>]
Jul 12 10:42:54 server courierdsn S E5D6FEFC 4224 2593 2588
2592 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0134d24>] [<c0158ab8>]
[<c0158bbb>] [<c014c10d>] [<c0109a3f>]
Jul 12 10:42:54 server sh S E71E8000 0 5024 1933 5027
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<c012e35d>] [<c0125c94>] [<c012d630>]
[<c0109a3f>]
Jul 12 10:42:54 server ping S F7332000 2388 5025 5024
5026 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa91a178>] [<fa918444>] [<fa91a190>]
[<fa91a190>] [<fa9185e1>]
Jul 12 10:42:54 server [<c014b628>] [<fa91906f>] [<c0109a3f>]
Jul 12 10:42:54 server grep S E71E8000 28 5026 5024
5027 5025 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa91a170>] [<fa9185e1>]
[<fa918e0a>] [<c0109a3f>]
Jul 12 10:42:54 server awk S F6D34000 2384 5027 5024
5026 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S E9AF4000 0 5028 2170 5029
5031 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01e3059>]
Jul 12 10:42:54 server [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F793A005 0 5029 5028
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server modprobe S F709E005 2388 5030 51
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c01193cc>] [<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S E81B6000 0 5031 2170 5032
5039 5028 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01e3059>]
Jul 12 10:42:54 server [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F709E005 0 5032 5031
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server fcron S F6FD2005 0 5033 2318
5035 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c013b161>] [<c0109a3f>]
Jul 12 10:42:54 server courieresmtp S F709E005 0 5034 2592
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server fcron S F6FD2005 8 5035 2318
5036 5033 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0107cad>] [<c0109a3f>]
Jul 12 10:42:54 server fcron S F6FD2005 2384 5036 2318
5038 5035 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0107cad>] [<c0109a3f>]
Jul 12 10:42:54 server fcron S F6FD2005 2384 5038 2318
5042 5036 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0107cad>] [<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S DFF4C000 2384 5039 2170 5040
5046 5031 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01e3059>]
Jul 12 10:42:54 server [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F709E005 4 5040 5039
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server couriertls S F6FD2005 2384 5041 2295
5048 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server fcron S F6FD2005 2384 5042 2318
5168 5038 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0107cad>] [<c0109a3f>]
Jul 12 10:42:54 server smbd S F6FD2013 2384 5043 2997
5044 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0135b21>] [<c0135f12>]
Jul 12 10:42:54 server [<c0109a3f>]
Jul 12 10:42:54 server smbd S F6FD2013 4 5044 2997
5045 5043 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0135b21>] [<c0135f12>]
Jul 12 10:42:54 server [<c0109a3f>]
Jul 12 10:42:54 server smbd S F6FD2013 0 5045 2997
5179 5044 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0135b21>] [<c0135f12>]
Jul 12 10:42:54 server [<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S E4C04000 8 5046 2170 5047
5049 5039 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01e3059>]
Jul 12 10:42:54 server [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F709E005 0 5047 5046
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server couriertls S F6FD2005 1716 5048 2295
5051 5041 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S E691A000 0 5049 2170 5050
5052 5046 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01e3059>]
Jul 12 10:42:54 server [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F709E005 328 5050 5049
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server couriertls S F709E005 424 5051 2295
5054 5048 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S E6C02000 0 5052 2170 5053
5170 5049 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01e3059>]
Jul 12 10:42:54 server [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F709E005 0 5053 5052
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server couriertls S F6FD2005 0 5054 2295
5051 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server fcron S F6FD2005 0 5168 2318
5169 5042 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0107cad>] [<c0109a3f>]
Jul 12 10:42:54 server fcron S F6FD2005 0 5169 2318
5172 5168 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0107cad>] [<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S DF478000 0 5170 2170 5171
5174 5052 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01e3059>]
Jul 12 10:42:54 server [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F709E005 0 5171 5170
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server fcron S F6FD2005 0 5172 2318
5173 5169 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0179815>] [<c0125e89>]
Jul 12 10:42:54 server [<c0107cad>] [<c0109a3f>]
Jul 12 10:42:54 server fcron S F6FD2005 0 5173 2318
5176 5172 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0107cad>] [<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S DF376000 0 5174 2170 5175
5182 5170 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01e3059>]
Jul 12 10:42:54 server [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F709E005 0 5175 5174
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server fcron S F709E005 0 5176 2318
5177 5173 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0107cad>] [<c0109a3f>]
Jul 12 10:42:54 server fcron S F709E005 0 5177 2318
5178 5176 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0107cad>] [<c0109a3f>]
Jul 12 10:42:54 server fcron S F709E005 2384 5178 2318
5177 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0107cad>] [<c0109a3f>]
Jul 12 10:42:54 server smbd S F6FD2013 2384 5179 2997
5180 5045 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0135b21>] [<c0135f12>]
Jul 12 10:42:54 server [<c0109a3f>]
Jul 12 10:42:54 server smbd S F6FD2013 0 5180 2997
5181 5179 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0135b21>] [<c0135f12>]
Jul 12 10:42:54 server [<c0109a3f>]
Jul 12 10:42:54 server smbd S F6FD2013 0 5181 2997
5322 5180 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>]
[<fa918e0a>] [<c0135b21>] [<c0135f12>]
Jul 12 10:42:54 server [<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S E0608000 2384 5182 2170 5183
5184 5174 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c01277c4>]
[<c01e3059>] [<c01e3ea4>]
Jul 12 10:42:54 server [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F709E005 0 5183 5182
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S E87B2000 2384 5184 2170 5185
5187 5182 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01e3059>]
Jul 12 10:42:54 server [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F709E005 0 5185 5184
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S E0320000 2384 5187 2170 5188
5189 5184 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01e3059>]
Jul 12 10:42:54 server [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F709E005 0 5188 5187
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S DFE8E000 4 5189 2170 5190
5310 5187 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01e3059>]
Jul 12 10:42:54 server [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F709E005 0 5190 5189
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c012b3d6>] [<c0109a3f>]
Jul 12 10:42:54 server httpd S F709E005 0 5309 3111
5848 16325 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0136067>] [<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S E00E2000 0 5310 2170 5311
5312 5189 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01e3059>]
Jul 12 10:42:54 server [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F709E005 2100 5311 5310
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c01193cc>] [<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S E55DE000 0 5312 2170 5313
5314 5310 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01e3059>]
Jul 12 10:42:54 server [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F709E005 0 5313 5312
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S E77D2000 0 5314 2170 5315
5316 5312 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01e3059>]
Jul 12 10:42:54 server [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F709E005 0 5315 5314
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S E085C000 0 5316 2170 5317
5318 5314 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01e3059>]
Jul 12 10:42:54 server [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F709E005 0 5317 5316
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S E8440000 0 5318 2170 5319
5320 5316 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01e3059>]
Jul 12 10:42:54 server [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F709E005 0 5319 5318
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S E7B5E000 2384 5320 2170 5321
5325 5318 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01e3059>]
Jul 12 10:42:54 server [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F709E005 0 5321 5320
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server smbd S F6FD2013 0 5322 2997
5323 5181 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0135b21>] [<c0135f12>]
Jul 12 10:42:54 server [<c0109a3f>]
Jul 12 10:42:54 server smbd S F6FD2013 0 5323 2997
5324 5322 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0135b21>] [<c0135f12>]
Jul 12 10:42:54 server [<c0109a3f>]
Jul 12 10:42:54 server smbd S F709E013 0 5324 2997
5453 5323 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0135b21>] [<c0135f12>]
Jul 12 10:42:54 server [<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S DF04C000 0 5325 2170 5326
5328 5320 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01e3059>]
Jul 12 10:42:54 server [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F709E005 0 5326 5325
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S EB504000 2384 5328 2170 5329
5446 5325 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01e3059>]
Jul 12 10:42:54 server [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F709E005 0 5329 5328
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S E414C000 0 5446 2170 5447
5448 5328 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01e3059>]
Jul 12 10:42:54 server [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F709E005 0 5447 5446
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S E0FEE000 4 5448 2170 5449
5450 5446 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01e3059>]
Jul 12 10:42:54 server [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F709E005 144 5449 5448
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S DFD58000 0 5450 2170 5451
5456 5448 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01e3059>]
Jul 12 10:42:54 server [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F709E005 0 5451 5450
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server smbd S F709E013 0 5453 2997
5454 5324 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0135b21>] [<c0135f12>]
Jul 12 10:42:54 server [<c0109a3f>]
Jul 12 10:42:54 server smbd S F709E013 0 5454 2997
5455 5453 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0135b21>] [<c0135f12>]
Jul 12 10:42:54 server [<c0109a3f>]
Jul 12 10:42:54 server smbd S F709E013 0 5455 2997
5591 5454 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0135b21>] [<c0135f12>]
Jul 12 10:42:54 server [<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S E09EC000 0 5456 2170 5457
5458 5450 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01e3059>]
Jul 12 10:42:54 server [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F709E005 0 5457 5456
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S E54CC000 952 5458 2170 5459
5460 5456 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01e3059>]
Jul 12 10:42:54 server [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F709E005 456 5459 5458
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S E5364000 0 5460 2170 5461
5462 5458 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01e3059>]
Jul 12 10:42:54 server [<c01e3ea4>] [<c011b680>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F709E005 0 5461 5460
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S DE93C000 0 5462 2170 5463
5464 5460 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01e3059>]
Jul 12 10:42:54 server [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F709E005 0 5463 5462
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S E4066000 0 5464 2170 5465
5466 5462 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01e3059>]
Jul 12 10:42:54 server [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F709E005 0 5465 5464
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S E8E4E000 2384 5466 2170 5467
5584 5464 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01e3059>]
Jul 12 10:42:54 server [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F709E005 0 5467 5466
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S E0A58000 0 5584 2170 5585
5587 5466 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01e3059>]
Jul 12 10:42:54 server [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F709E005 0 5585 5584
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S E1336000 0 5587 2170 5588
5589 5584 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01e3059>]
Jul 12 10:42:54 server [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F6FD2005 0 5588 5587
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c012b3d6>] [<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S DEB2A000 0 5589 2170 5590
5711 5587 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01277c4>]
Jul 12 10:42:54 server [<c01e3059>] [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F6FD2005 0 5590 5589
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server smbd S F709E013 0 5591 2997
5592 5455 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0135b21>] [<c0135f12>]
Jul 12 10:42:54 server [<c0109a3f>]
Jul 12 10:42:54 server smbd S F709E013 0 5592 2997
5593 5591 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0135b21>] [<c0135f12>]
Jul 12 10:42:54 server [<c0109a3f>]
Jul 12 10:42:54 server smbd S F709E013 0 5593 2997
5728 5592 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0135b21>] [<c0135f12>]
Jul 12 10:42:54 server [<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S E5410000 0 5711 2170 5712
5713 5589 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01277c4>]
Jul 12 10:42:54 server [<c01e3059>] [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F6FD2005 0 5712 5711
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S DF7FE000 0 5713 2170 5714
5715 5711 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01e3059>]
Jul 12 10:42:54 server [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F6FD2005 0 5714 5713
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S DFE1A000 4 5715 2170 5716
5717 5713 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01e3059>]
Jul 12 10:42:54 server [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F709E005 4 5716 5715
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S E40E8000 2384 5717 2170 5718
5719 5715 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01e3059>]
Jul 12 10:42:54 server [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F6FD2005 0 5718 5717
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S DFC1A000 0 5719 2170 5720
5721 5717 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01277c4>]
Jul 12 10:42:54 server [<c01e3059>] [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F709E005 0 5720 5719
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S DF664000 0 5721 2170 5722
5723 5719 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01e3059>]
Jul 12 10:42:54 server [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F6FD2005 0 5722 5721
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S E13BA000 0 5723 2170 5724
5725 5721 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c01277c4>]
[<c01e3059>] [<c01e3ea4>]
Jul 12 10:42:54 server [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F6FD2005 0 5724 5723
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server couriertcpd S C02CF800 0 5725 2170 5726
5723 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<c0125c94>] [<c0107d1d>] [<c0125fa7>]
[<c0109a3f>] [<c0130f5f>]
Jul 12 10:42:54 server [<c0130d80>] [<c023ac0f>] [<c01e2fd8>] [<c0130030>]
[<c0138b40>] [<c01e3059>]
Jul 12 10:42:54 server [<c01e3ea4>] [<c0109a3f>]
Jul 12 10:42:54 server modprobe S F6FD2005 4 5726 5725
(NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0109a3f>]
Jul 12 10:42:54 server smbd S F709E013 2384 5728 2997
5729 5593 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0135b21>] [<c0135f12>]
Jul 12 10:42:54 server [<c0109a3f>]
Jul 12 10:42:54 server smbd S F709E013 0 5729 2997
5730 5728 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0135b21>] [<c0135f12>]
Jul 12 10:42:54 server [<c0109a3f>]
Jul 12 10:42:54 server smbd S F709E013 0 5730 2997
5729 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c0135b21>] [<c0135f12>]
Jul 12 10:42:54 server [<c0109a3f>]
Jul 12 10:42:54 server httpd S F6FD2005 0 5848 3111
5309 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa9185e1>] [<fa918e0a>]
[<c012b3d6>] [<c0109a3f>]
Jul 12 10:42:54 server shutdown S F709E005 0 5856 1
1964 (NOTLB)
Jul 12 10:42:54 server Call Trace: [<fa9182a9>] [<fa91a170>] [<fa9185e1>]
[<fa918e0a>] [<c0109a3f>]
Startup
Jul 12 01:06:33 server syslog-ng[246]: syslog-ng version 1.6.0rc3 going down
Jul 12 01:07:12 server syslog-ng[245]: syslog-ng version 1.6.0rc3 starting
Sign of the problem
Jul 12 07:12:51 server fcron[4063]: Job /etc/zorbiptraffic/zorbiptraffic.pl
> /dev/null started for user root (pid 4064) (This one worked)
Jul 12 07:32:51 server fcron[2318]: process already running: root's
/etc/zorbiptraffic/zorbiptraffic.pl > /dev/null
Jul 12 08:02:51 server fcron[2318]: process already running: root's
/etc/zorbiptraffic/zorbiptraffic.pl > /dev/null
Jul 12 08:30:00 server fcron[2318]: process already running: jim's
/usr/bin/fetchmail
Jul 12 08:32:00 server fcron[2318]: process already running: annie's
/usr/bin/fetchmail
Restart
Jul 12 10:43:38 server syslog-ng[245]: syslog-ng version 1.6.0rc3 starting
Jul 12 10:43:38 server syslog-ng[245]: Changing permissions on special file
/dev/tty12
Everything working
Jul 12 10:45:10 server fcron[2462]: Job /usr/bin/fetchmail started for user
annie (pid 2463)
Jul 12 10:45:40 server fcron[2462]: Job /usr/bin/fetchmail terminated (exit
status: 1)
Jul 12 10:45:41 server fcron[2515]: Job /usr/bin/fetchmail started for user
annie (pid 2516)
Jul 12 10:47:28 server fcron[3606]: Job /etc/zorbiptraffic/zorbiptraffic.pl
> /dev/null started for user root (pid 3607)
Jul 12 10:49:14 server fcron[3653]: Job /etc/zorbiptraffic/zorbiptraffic.pl
> /dev/null started for user root (pid 3654)
Jul 12 11:00:02 server fcron[4270]: Job /etc/zorbiptraffic/zorbiptraffic.pl
> /dev/null started for user root (pid 4272)
Jul 12 11:00:02 server fcron[4271]: Job /usr/bin/fetchmail started for user
jim (pid 4273)
Jul 12 11:00:13 server fcron[4271]: Job /usr/bin/fetchmail terminated (exit
status: 1)
ver_linux
Linux server 2.4.22-pre5 #1 SMP Fri Jul 11 23:38:00 PDT 2003 i686 unknown
unknown GNU/Linux
Gnu C 3.3
Gnu make 3.80
util-linux 2.11z
mount 2.11z
modutils 2.4.25
e2fsprogs 1.33
PPP 2.4.1
Linux C Library 2.3.2
Dynamic linker (ldd) 2.3.2
Linux C++ Library 5.0.4
Procps 3.1.9
Net-tools 1.60
Kbd 1.08
Sh-utils 5.0
Modules Loaded dazuko ipt_TCPMSS ipt_TOS ipt_string ipt_psd
ipt_pkttype ipt_unclean ipt_state ipt_REJECT ipt_LOG ipt_limit
iptable_mangle iptable_nat iptable_filter ip_tables ip_conntrack_h323
ip_conntrack_mms ip_conntrack_irc ip_conntrack_ftp ip_conntrack tulip rtc
usb-ohci usbcore aic7xxx megaraid sd_mod scsi_mod
My config is attached.
----- Original Message -----
From: "Marcelo Tosatti" <[email protected]>
To: "Jim Gifford" <[email protected]>
Cc: "lkml" <[email protected]>
Sent: Friday, July 11, 2003 1:49 PM
Subject: Re: Linux 2.4.22-pre3
>
>
> On Fri, 11 Jul 2003, Jim Gifford wrote:
>
> > Ok,
> > Here is what I got from my check. Snort is the culprit. When it
starts,
> > it runs at 40mb. Each hour it adds 20 mb. So by the time the system is
up
> > for hours, it starts cosuming memory. Today after I restartd my system,
I
> > noticed every hour it was adding a mb of RAM.
> >
> > It started at 40mb and after 4 hours is up to 45mb.
> >
> > Thanx all for your help and assistance.
>
> Ok, thanks for your report.
>
> Try -pre5 8)
>
On Sat, 12 Jul 2003, Jim Gifford wrote:
> I started linux-2.4.22-pre5 at 1:00am it stopped at 8:00am, with no error
> messages. I did enable sysrq keys here is the information.
Jim,
Can you please use ksymoops and your System.map to convert the backtraces
to human readable format?
You're not using quota, so you are not hitting the quota/ext3 deadlock.
As requested.
----- Original Message -----
From: "Marcelo Tosatti" <[email protected]>
To: "Jim Gifford" <[email protected]>
Cc: "lkml" <[email protected]>
Sent: Monday, July 14, 2003 7:48 AM
Subject: 2.4.22-pre5 deadlock
>
>
> On Sat, 12 Jul 2003, Jim Gifford wrote:
>
> > I started linux-2.4.22-pre5 at 1:00am it stopped at 8:00am, with no
error
> > messages. I did enable sysrq keys here is the information.
>
> Jim,
>
> Can you please use ksymoops and your System.map to convert the backtraces
> to human readable format?
>
> You're not using quota, so you are not hitting the quota/ext3 deadlock.
>
>
----- Original Message -----
From: "Marcelo Tosatti" <[email protected]>
To: "Jim Gifford" <[email protected]>
Cc: "lkml" <[email protected]>
Sent: Monday, July 14, 2003 7:48 AM
Subject: 2.4.22-pre5 deadlock
>
>
> On Sat, 12 Jul 2003, Jim Gifford wrote:
>
> > I started linux-2.4.22-pre5 at 1:00am it stopped at 8:00am, with no
error
> > messages. I did enable sysrq keys here is the information.
>
> Jim,
>
> Can you please use ksymoops and your System.map to convert the backtraces
> to human readable format?
>
> You're not using quota, so you are not hitting the quota/ext3 deadlock.
>
>
Any ideas on what I should be looking for. I sent the backtraces a few days
back, pre6 just locked up after 20 hours. For the heck of things, I logged
the building of the kernel, I didn't notice to many problems, I have
attached the build log figuring it may help decipher the problem.
On Mon, Jul 14, 2003 at 10:03:03AM -0700, Jim Gifford wrote:
> As requested.
please try to reproduce w/o devfs and/or w/o a kernel module that is
loadable called ipt_psd (netfilter stuff likely, but not part of
mainline pre6/pre7). probably it'll go away either ways and it seems
triggered by the process called couriertcpd. Not sure exactly what's
going on though, since looking into devfs/devfsd doesn't sound
interesting anymore and I don't see the netfilter code out of mainline.
(probably this email will get some delay, so apologies if it is obsolete
by the time it reaches the network)
Andrea
----- Original Message -----
From: "Andrea Arcangeli" <[email protected]>
To: "Jim Gifford" <[email protected]>
Cc: "Marcelo Tosatti" <[email protected]>; "lkml"
<[email protected]>
Sent: Saturday, July 19, 2003 10:21 AM
Subject: Re: 2.4.22-pre5 deadlock
> On Mon, Jul 14, 2003 at 10:03:03AM -0700, Jim Gifford wrote:
> > As requested.
>
> please try to reproduce w/o devfs and/or w/o a kernel module that is
> loadable called ipt_psd (netfilter stuff likely, but not part of
> mainline pre6/pre7). probably it'll go away either ways and it seems
> triggered by the process called couriertcpd. Not sure exactly what's
> going on though, since looking into devfs/devfsd doesn't sound
> interesting anymore and I don't see the netfilter code out of mainline.
>
> (probably this email will get some delay, so apologies if it is obsolete
> by the time it reaches the network)
>
> Andrea
>
I have removed all non-standard iptables modules and the dazuko module. It
locked up under pre6 without these modules, but pre7 hasn't caused a problem
yet, it's at 28 hours so far, the record is three days. I have noticed
increased memory usage, which is the starting sign of problems.
On Sun, 20 Jul 2003, Jim Gifford wrote:
> ----- Original Message -----
> From: "Andrea Arcangeli" <[email protected]>
> To: "Jim Gifford" <[email protected]>
> Cc: "Marcelo Tosatti" <[email protected]>; "lkml"
> <[email protected]>
> Sent: Saturday, July 19, 2003 10:21 AM
> Subject: Re: 2.4.22-pre5 deadlock
>
>
> > On Mon, Jul 14, 2003 at 10:03:03AM -0700, Jim Gifford wrote:
> > > As requested.
> >
> > please try to reproduce w/o devfs and/or w/o a kernel module that is
> > loadable called ipt_psd (netfilter stuff likely, but not part of
> > mainline pre6/pre7). probably it'll go away either ways and it seems
> > triggered by the process called couriertcpd. Not sure exactly what's
> > going on though, since looking into devfs/devfsd doesn't sound
> > interesting anymore and I don't see the netfilter code out of mainline.
> >
> > (probably this email will get some delay, so apologies if it is obsolete
> > by the time it reaches the network)
> >
> > Andrea
> >
> I have removed all non-standard iptables modules and the dazuko module. It
> locked up under pre6 without these modules, but pre7 hasn't caused a problem
> yet, it's at 28 hours so far, the record is three days. I have noticed
> increased memory usage, which is the starting sign of problems.
Jim,
The increased memory usage seems normal. Its the kernel cahcing more and
more buffers: they will be freed when the system needs memory.
Lets wait and see what happens without the iptables and dazuko modules.
On Mon, Jul 21, 2003 at 09:45:32AM -0300, Marcelo Tosatti wrote:
>
>
> On Sun, 20 Jul 2003, Jim Gifford wrote:
>
> > ----- Original Message -----
> > From: "Andrea Arcangeli" <[email protected]>
> > To: "Jim Gifford" <[email protected]>
> > Cc: "Marcelo Tosatti" <[email protected]>; "lkml"
> > <[email protected]>
> > Sent: Saturday, July 19, 2003 10:21 AM
> > Subject: Re: 2.4.22-pre5 deadlock
> >
> >
> > > On Mon, Jul 14, 2003 at 10:03:03AM -0700, Jim Gifford wrote:
> > > > As requested.
> > >
> > > please try to reproduce w/o devfs and/or w/o a kernel module that is
> > > loadable called ipt_psd (netfilter stuff likely, but not part of
> > > mainline pre6/pre7). probably it'll go away either ways and it seems
> > > triggered by the process called couriertcpd. Not sure exactly what's
> > > going on though, since looking into devfs/devfsd doesn't sound
> > > interesting anymore and I don't see the netfilter code out of mainline.
> > >
> > > (probably this email will get some delay, so apologies if it is obsolete
> > > by the time it reaches the network)
> > >
> > > Andrea
> > >
> > I have removed all non-standard iptables modules and the dazuko module. It
> > locked up under pre6 without these modules, but pre7 hasn't caused a problem
> > yet, it's at 28 hours so far, the record is three days. I have noticed
> > increased memory usage, which is the starting sign of problems.
>
> Jim,
>
> The increased memory usage seems normal. Its the kernel cahcing more and
> more buffers: they will be freed when the system needs memory.
>
> Lets wait and see what happens without the iptables and dazuko modules.
also please w/o devfs, a task was hanging waiting for devfsd and there
were many request_module running from kmod, so it maybe a devfs/devfsd
race too (ipc_psd, may only be triggering it, and it may not be the
culprit).
Andrea
----- Original Message -----
From: "Marcelo Tosatti" <[email protected]>
To: "Jim Gifford" <[email protected]>
Cc: "Andrea Arcangeli" <[email protected]>; "lkml"
<[email protected]>
Sent: Monday, July 21, 2003 5:45 AM
Subject: Re: 2.4.22-pre5 deadlock
>
>
> On Sun, 20 Jul 2003, Jim Gifford wrote:
>
> > ----- Original Message -----
> > From: "Andrea Arcangeli" <[email protected]>
> > To: "Jim Gifford" <[email protected]>
> > Cc: "Marcelo Tosatti" <[email protected]>; "lkml"
> > <[email protected]>
> > Sent: Saturday, July 19, 2003 10:21 AM
> > Subject: Re: 2.4.22-pre5 deadlock
> >
> >
> > > On Mon, Jul 14, 2003 at 10:03:03AM -0700, Jim Gifford wrote:
> > > > As requested.
> > >
> > > please try to reproduce w/o devfs and/or w/o a kernel module that is
> > > loadable called ipt_psd (netfilter stuff likely, but not part of
> > > mainline pre6/pre7). probably it'll go away either ways and it seems
> > > triggered by the process called couriertcpd. Not sure exactly what's
> > > going on though, since looking into devfs/devfsd doesn't sound
> > > interesting anymore and I don't see the netfilter code out of
mainline.
> > >
> > > (probably this email will get some delay, so apologies if it is
obsolete
> > > by the time it reaches the network)
> > >
> > > Andrea
> > >
> > I have removed all non-standard iptables modules and the dazuko module.
It
> > locked up under pre6 without these modules, but pre7 hasn't caused a
problem
> > yet, it's at 28 hours so far, the record is three days. I have noticed
> > increased memory usage, which is the starting sign of problems.
>
> Jim,
>
> The increased memory usage seems normal. Its the kernel cahcing more and
> more buffers: they will be freed when the system needs memory.
>
> Lets wait and see what happens without the iptables and dazuko modules.
>
This could be revelant to my situation
http://lists.netfilter.org/pipermail/netfilter/2003-July/045504.html
----- Original Message -----
From: "Marcelo Tosatti" <[email protected]>
To: "Jim Gifford" <[email protected]>
Cc: "Andrea Arcangeli" <[email protected]>; "lkml"
<[email protected]>
Sent: Monday, July 21, 2003 5:45 AM
Subject: Re: 2.4.22-pre5 deadlock
>
>
> On Sun, 20 Jul 2003, Jim Gifford wrote:
>
> > ----- Original Message -----
> > From: "Andrea Arcangeli" <[email protected]>
> > To: "Jim Gifford" <[email protected]>
> > Cc: "Marcelo Tosatti" <[email protected]>; "lkml"
> > <[email protected]>
> > Sent: Saturday, July 19, 2003 10:21 AM
> > Subject: Re: 2.4.22-pre5 deadlock
> >
> >
> > > On Mon, Jul 14, 2003 at 10:03:03AM -0700, Jim Gifford wrote:
> > > > As requested.
> > >
> > > please try to reproduce w/o devfs and/or w/o a kernel module that is
> > > loadable called ipt_psd (netfilter stuff likely, but not part of
> > > mainline pre6/pre7). probably it'll go away either ways and it seems
> > > triggered by the process called couriertcpd. Not sure exactly what's
> > > going on though, since looking into devfs/devfsd doesn't sound
> > > interesting anymore and I don't see the netfilter code out of
mainline.
> > >
> > > (probably this email will get some delay, so apologies if it is
obsolete
> > > by the time it reaches the network)
> > >
> > > Andrea
> > >
> > I have removed all non-standard iptables modules and the dazuko module.
It
> > locked up under pre6 without these modules, but pre7 hasn't caused a
problem
> > yet, it's at 28 hours so far, the record is three days. I have noticed
> > increased memory usage, which is the starting sign of problems.
>
> Jim,
>
> The increased memory usage seems normal. Its the kernel cahcing more and
> more buffers: they will be freed when the system needs memory.
>
> Lets wait and see what happens without the iptables and dazuko modules.
>
Marcelo,
-pre7 seems to be working ok. Do you want me to enable the dazuko thing
again to see if it's the cause, or do you want me to wait a little longer to
see what happens.
----- Original Message -----
From: "Marcelo Tosatti" <[email protected]>
To: "Jim Gifford" <[email protected]>
Cc: "Andrea Arcangeli" <[email protected]>; "lkml"
<[email protected]>
Sent: Monday, July 21, 2003 5:45 AM
Subject: Re: 2.4.22-pre5 deadlock
>
>
> On Sun, 20 Jul 2003, Jim Gifford wrote:
>
> > ----- Original Message -----
> > From: "Andrea Arcangeli" <[email protected]>
> > To: "Jim Gifford" <[email protected]>
> > Cc: "Marcelo Tosatti" <[email protected]>; "lkml"
> > <[email protected]>
> > Sent: Saturday, July 19, 2003 10:21 AM
> > Subject: Re: 2.4.22-pre5 deadlock
> >
> >
> > > On Mon, Jul 14, 2003 at 10:03:03AM -0700, Jim Gifford wrote:
> > > > As requested.
> > >
> > > please try to reproduce w/o devfs and/or w/o a kernel module that is
> > > loadable called ipt_psd (netfilter stuff likely, but not part of
> > > mainline pre6/pre7). probably it'll go away either ways and it seems
> > > triggered by the process called couriertcpd. Not sure exactly what's
> > > going on though, since looking into devfs/devfsd doesn't sound
> > > interesting anymore and I don't see the netfilter code out of
mainline.
> > >
> > > (probably this email will get some delay, so apologies if it is
obsolete
> > > by the time it reaches the network)
> > >
> > > Andrea
> > >
> > I have removed all non-standard iptables modules and the dazuko module.
It
> > locked up under pre6 without these modules, but pre7 hasn't caused a
problem
> > yet, it's at 28 hours so far, the record is three days. I have noticed
> > increased memory usage, which is the starting sign of problems.
>
> Jim,
>
> The increased memory usage seems normal. Its the kernel cahcing more and
> more buffers: they will be freed when the system needs memory.
>
> Lets wait and see what happens without the iptables and dazuko modules.
>
People on the Dell Boards keep talking about Redhat 8.0 locking up.
http://forums.us.dell.com/supportforums/board/message?board.id=pes_linux&mes
sage.id=1224
On Mon, 21 Jul 2003, Jim Gifford wrote:
> > Lets wait and see what happens without the iptables and dazuko modules.
> >
> Marcelo,
> -pre7 seems to be working ok. Do you want me to enable the dazuko thing
> again to see if it's the cause, or do you want me to wait a little longer to
> see what happens.
Jim,
I prefer if you leave -pre7 running for a while to confirm its stable.
----- Original Message -----
From: "Marcelo Tosatti" <[email protected]>
To: "Jim Gifford" <[email protected]>
Cc: "Andrea Arcangeli" <[email protected]>; "lkml"
<[email protected]>
Sent: Tuesday, July 22, 2003 4:53 AM
Subject: Re: 2.4.22-pre5 deadlock
>
>
> On Mon, 21 Jul 2003, Jim Gifford wrote:
>
> > > Lets wait and see what happens without the iptables and dazuko
modules.
> > >
> > Marcelo,
> > -pre7 seems to be working ok. Do you want me to enable the dazuko
thing
> > again to see if it's the cause, or do you want me to wait a little
longer to
> > see what happens.
>
> Jim,
>
> I prefer if you leave -pre7 running for a while to confirm its stable.
>
>
>
top - 08:29:37 up 2 days, 13:41, 2 users, load average: 2.33, 2.12, 2.03
Tasks: 109 total, 3 running, 106 sleeping, 0 stopped, 0 zombie
Cpu0 : 0.3% user, 3.9% system, 95.8% nice, 0.0% idle
Cpu1 : 1.3% user, 0.6% system, 98.1% nice, 0.0% idle
Mem: 1033672k total, 639904k used, 393768k free, 160764k buffers
Swap: 265060k total, 0k used, 265060k free, 187444k cached
Do you want me to do something intensive. I have a compile that I can do
that takes over 8 hours.
On Tue, 2003-07-22 at 02:40, Jim Gifford wrote:
> >
> People on the Dell Boards keep talking about Redhat 8.0 locking up.
> http://forums.us.dell.com/supportforums/board/message?board.id=pes_linux&mes
> sage.id=1224
Dell PE2650 lock up without a recent (minimum 1.4c) tg3 ethernet driver.
-tony
On Tue, 22 Jul 2003, Jim Gifford wrote:
> ----- Original Message -----
> From: "Marcelo Tosatti" <[email protected]>
> To: "Jim Gifford" <[email protected]>
> Cc: "Andrea Arcangeli" <[email protected]>; "lkml"
> <[email protected]>
> Sent: Tuesday, July 22, 2003 4:53 AM
> Subject: Re: 2.4.22-pre5 deadlock
>
>
> >
> >
> > On Mon, 21 Jul 2003, Jim Gifford wrote:
> >
> > > > Lets wait and see what happens without the iptables and dazuko
> modules.
> > > >
> > > Marcelo,
> > > -pre7 seems to be working ok. Do you want me to enable the dazuko
> thing
> > > again to see if it's the cause, or do you want me to wait a little
> longer to
> > > see what happens.
> >
> > Jim,
> >
> > I prefer if you leave -pre7 running for a while to confirm its stable.
> >
> >
> >
> top - 08:29:37 up 2 days, 13:41, 2 users, load average: 2.33, 2.12, 2.03
> Tasks: 109 total, 3 running, 106 sleeping, 0 stopped, 0 zombie
> Cpu0 : 0.3% user, 3.9% system, 95.8% nice, 0.0% idle
> Cpu1 : 1.3% user, 0.6% system, 98.1% nice, 0.0% idle
> Mem: 1033672k total, 639904k used, 393768k free, 160764k buffers
> Swap: 265060k total, 0k used, 265060k free, 187444k cached
>
> Do you want me to do something intensive. I have a compile that I can do
> that takes over 8 hours.
Jim,
I guess most of us is already convinced that the lockups were caused by
the non-stock code.
How long it usually took to lockup before?
----- Original Message -----
From: "Marcelo Tosatti" <[email protected]>
To: "Jim Gifford" <[email protected]>
Cc: "Andrea Arcangeli" <[email protected]>; "lkml"
<[email protected]>
Sent: Tuesday, July 22, 2003 10:00 AM
Subject: Re: 2.4.22-pre5 deadlock
>
>
> On Tue, 22 Jul 2003, Jim Gifford wrote:
>
> > ----- Original Message -----
> > From: "Marcelo Tosatti" <[email protected]>
> > To: "Jim Gifford" <[email protected]>
> > Cc: "Andrea Arcangeli" <[email protected]>; "lkml"
> > <[email protected]>
> > Sent: Tuesday, July 22, 2003 4:53 AM
> > Subject: Re: 2.4.22-pre5 deadlock
> >
> >
> > >
> > >
> > > On Mon, 21 Jul 2003, Jim Gifford wrote:
> > >
> > > > > Lets wait and see what happens without the iptables and dazuko
> > modules.
> > > > >
> > > > Marcelo,
> > > > -pre7 seems to be working ok. Do you want me to enable the
dazuko
> > thing
> > > > again to see if it's the cause, or do you want me to wait a little
> > longer to
> > > > see what happens.
> > >
> > > Jim,
> > >
> > > I prefer if you leave -pre7 running for a while to confirm its stable.
> > >
> > >
> > >
> > top - 08:29:37 up 2 days, 13:41, 2 users, load average: 2.33, 2.12,
2.03
> > Tasks: 109 total, 3 running, 106 sleeping, 0 stopped, 0 zombie
> > Cpu0 : 0.3% user, 3.9% system, 95.8% nice, 0.0% idle
> > Cpu1 : 1.3% user, 0.6% system, 98.1% nice, 0.0% idle
> > Mem: 1033672k total, 639904k used, 393768k free, 160764k buffers
> > Swap: 265060k total, 0k used, 265060k free, 187444k cached
> >
> > Do you want me to do something intensive. I have a compile that I can do
> > that takes over 8 hours.
>
> Jim,
>
> I guess most of us is already convinced that the lockups were caused by
> the non-stock code.
>
> How long it usually took to lockup before?
>
I removed the modules and dazuko no pre6 and it locked up. But pre7 seems to
be running great, I'll try adding the dazuko module first, and see what
happens.
Latest Info
top - 18:03:30 up 2 days, 23:15, 2 users, load average: 2.14, 2.15, 2.05
Tasks: 108 total, 3 running, 105 sleeping, 0 stopped, 0 zombie
Cpu0 : 0.0% user, 1.3% system, 98.4% nice, 0.3% idle
Cpu1 : 1.9% user, 2.9% system, 95.2% nice, 0.0% idle
Mem: 1033672k total, 688872k used, 344800k free, 164496k buffers
Swap: 265060k total, 0k used, 265060k free, 211988k cached
I will add dazuko in and report back.
----- Original Message -----
From: "Marcelo Tosatti" <[email protected]>
To: "Jim Gifford" <[email protected]>
Cc: "Andrea Arcangeli" <[email protected]>; "lkml"
<[email protected]>
Sent: Tuesday, July 22, 2003 10:00 AM
Subject: Re: 2.4.22-pre5 deadlock
>
>
> On Tue, 22 Jul 2003, Jim Gifford wrote:
>
> > ----- Original Message -----
> > From: "Marcelo Tosatti" <[email protected]>
> > To: "Jim Gifford" <[email protected]>
> > Cc: "Andrea Arcangeli" <[email protected]>; "lkml"
> > <[email protected]>
> > Sent: Tuesday, July 22, 2003 4:53 AM
> > Subject: Re: 2.4.22-pre5 deadlock
> >
> >
> > >
> > >
> > > On Mon, 21 Jul 2003, Jim Gifford wrote:
> > >
> > > > > Lets wait and see what happens without the iptables and dazuko
> > modules.
> > > > >
> > > > Marcelo,
> > > > -pre7 seems to be working ok. Do you want me to enable the
dazuko
> > thing
> > > > again to see if it's the cause, or do you want me to wait a little
> > longer to
> > > > see what happens.
> > >
> > > Jim,
> > >
> > > I prefer if you leave -pre7 running for a while to confirm its stable.
> > >
> > >
> > >
> > top - 08:29:37 up 2 days, 13:41, 2 users, load average: 2.33, 2.12,
2.03
> > Tasks: 109 total, 3 running, 106 sleeping, 0 stopped, 0 zombie
> > Cpu0 : 0.3% user, 3.9% system, 95.8% nice, 0.0% idle
> > Cpu1 : 1.3% user, 0.6% system, 98.1% nice, 0.0% idle
> > Mem: 1033672k total, 639904k used, 393768k free, 160764k buffers
> > Swap: 265060k total, 0k used, 265060k free, 187444k cached
> >
> > Do you want me to do something intensive. I have a compile that I can do
> > that takes over 8 hours.
>
> Jim,
>
> I guess most of us is already convinced that the lockups were caused by
> the non-stock code.
>
> How long it usually took to lockup before?
>
I have the dazuko thing running with it's application. Still up after 19
hours. So something in -pre7 seems to have fixed the problem. Last time we
tested pre6 with the dazuko thing it deadlocked, I still haven't enabled the
rest of the iptables options yet. I will keep it running a till tomorrow and
see what happens.
>From talking with others, we are considering this a netfilter issue, is this
correct??
On Fri, 25 Jul 2003, Jim Gifford wrote:
> >From talking with others, we are considering this a netfilter issue, is this
> correct??
It seems you have isolated the problem down to additional netfilter
patches right ?
----- Original Message -----
From: "Marcelo Tosatti" <[email protected]>
To: "Jim Gifford" <[email protected]>
Cc: "Andrea Arcangeli" <[email protected]>; "lkml"
<[email protected]>; "Marc Heckmann" <[email protected]>
Sent: Friday, July 25, 2003 12:39 PM
Subject: Re: 2.4.22-pre5 deadlock
>
>
> On Fri, 25 Jul 2003, Jim Gifford wrote:
>
> > >From talking with others, we are considering this a netfilter issue, is
this
> > correct??
>
> It seems you have isolated the problem down to additional netfilter
> patches right ?
>
That is correct. The wierd part is that people are having problems with
different modules and it's hard to track down what is in common. It seems
the LOG and limit are the only common ones in the group. When I get back
from vacation, I'm going to add all the modules in except for LOG and limit
and see if the problem comes back.
On Fri, Jul 25, 2003 at 04:39:29PM -0300, Marcelo Tosatti wrote:
>
>
> On Fri, 25 Jul 2003, Jim Gifford wrote:
>
> > >From talking with others, we are considering this a netfilter issue, is this
> > correct??
>
> It seems you have isolated the problem down to additional netfilter
> patches right ?
no. standard netfilter modules. I just happened to notice that everyone
who had the problem seemed to be using
iptables.
just to recap what the problem is:
kernel 2.4.2[10] maybe even .19, etc SMP deadlock. systems still
responds to ping, iptables logs still scroll along on the console.
but nothing else.
please see my post with ksymoops output of sysrq+T :
http://marc.theaimsgroup.com/?l=linux-kernel&m=105821579624140&w=2
however, the deadlock happened when the system was under heavy disk IO.
all filesytems are raid1, ext3.
here are my modules:
[mh@zara mh]$ /sbin/lsmod
Module Size Used by Not tainted
ip_conntrack_ftp 5424 0 (unused)
8139too 17832 1
mii 4124 0 [8139too]
e100 55236 1
ipt_REJECT 4024 2 (autoclean)
ipt_state 1048 34 (autoclean)
ipt_limit 1688 2 (autoclean)
ipt_LOG 4248 4 (autoclean)
iptable_nat 22872 0 (autoclean) (unused)
ip_conntrack 29896 3 (autoclean) [ip_conntrack_ftp ipt_state
iptable_nat]
iptable_filter 2412 1 (autoclean)
ip_tables 15672 8 [ipt_REJECT ipt_state ipt_limit ipt_LOG
iptable_nat iptable_filter]
keybdev 2944 0 (unused)
mousedev 5688 0 (unused)
hid 16296 0 (unused)
input 6144 0 [keybdev mousedev hid]
usbcore 80704 1 [hid]
ext3 76096 3
jbd 65904 3 [ext3]
raid1 16108 4
-m
----- Original Message -----
From: "Marcelo Tosatti" <[email protected]>
To: "Jim Gifford" <[email protected]>
Cc: "Andrea Arcangeli" <[email protected]>; "lkml"
<[email protected]>
Sent: Tuesday, July 22, 2003 10:00 AM
Subject: Re: 2.4.22-pre5 deadlock
>
>
> On Tue, 22 Jul 2003, Jim Gifford wrote:
>
> > ----- Original Message -----
> > From: "Marcelo Tosatti" <[email protected]>
> > To: "Jim Gifford" <[email protected]>
> > Cc: "Andrea Arcangeli" <[email protected]>; "lkml"
> > <[email protected]>
> > Sent: Tuesday, July 22, 2003 4:53 AM
> > Subject: Re: 2.4.22-pre5 deadlock
> >
> >
> > >
> > >
> > > On Mon, 21 Jul 2003, Jim Gifford wrote:
> > >
> > > > > Lets wait and see what happens without the iptables and dazuko
> > modules.
> > > > >
> > > > Marcelo,
> > > > -pre7 seems to be working ok. Do you want me to enable the
dazuko
> > thing
> > > > again to see if it's the cause, or do you want me to wait a little
> > longer to
> > > > see what happens.
> > >
> > > Jim,
> > >
> > > I prefer if you leave -pre7 running for a while to confirm its stable.
> > >
> > >
> > >
> > top - 08:29:37 up 2 days, 13:41, 2 users, load average: 2.33, 2.12,
2.03
> > Tasks: 109 total, 3 running, 106 sleeping, 0 stopped, 0 zombie
> > Cpu0 : 0.3% user, 3.9% system, 95.8% nice, 0.0% idle
> > Cpu1 : 1.3% user, 0.6% system, 98.1% nice, 0.0% idle
> > Mem: 1033672k total, 639904k used, 393768k free, 160764k buffers
> > Swap: 265060k total, 0k used, 265060k free, 187444k cached
> >
> > Do you want me to do something intensive. I have a compile that I can do
> > that takes over 8 hours.
>
> Jim,
>
> I guess most of us is already convinced that the lockups were caused by
> the non-stock code.
>
> How long it usually took to lockup before?
>
Marcelo,
I just got back from vacation, pre8 was running with the dazuko module
and the program that used it. No down time at all, I will start adding the
iptables modules and see which one causes the problems. (I left on Sunday
8am returned on 8pm on Friday).
Jim