On Wed, Oct 01, 2008 at 12:23:58AM +0200, Quentin wrote:
>
> Of course. However since I unmounted and remounted /home the 'buffer' line
> is now only 59megs, and they are still not dropped when a program tries to
> malloc all the memory. I'll tell next time the problem shows up (it
> can take ten days)
>
Are you willing to patch and recompile your kernel? If so, the
following patch would be very helpful in determining what is going on.
It allows us to see what buffer heads are in use for a particular
block device. Attached please the kernel patch and the user program.
- Ted
P.S. Unfortunately, all of the code to debug buffer head leaks was
dropped when the buffer cache was moved into the page cache. Any
comments about a refined version of patch getting merged into the
mainline kernel as a debugging measure?
On Thu, Oct 02, 2008 at 08:35:48PM -0400, Theodore Tso wrote:
> On Wed, Oct 01, 2008 at 12:23:58AM +0200, Quentin wrote:
> >
> > Of course. However since I unmounted and remounted /home the 'buffer' line
> > is now only 59megs, and they are still not dropped when a program tries to
> > malloc all the memory. I'll tell next time the problem shows up (it
> > can take ten days)
> >
>
> Are you willing to patch and recompile your kernel? If so, the
> following patch would be very helpful in determining what is going on.
> It allows us to see what buffer heads are in use for a particular
> block device. Attached please the kernel patch and the user program.
Now that the machine is again in the 100M+ in buffers (still unreleased when
a program asks for all the memory), I launched the program on the devices
which support / and /home.
I also attached /proc/meminfo and /proc/slabinfo
In both cases it freezes solid the machine for more than a minute or so, and
it overflows the dmesg with messages. Here is shown what I got
[166632.382632] buffer dirty: block 35491 count 1
[166632.386827] buffer dirty: block 35493 count 3
[166632.391019] buffer dirty: block 35494 count 1
[166632.395251] buffer dirty: block 35496 count 1
[166632.399446] buffer dirty: block 35497 count 2
[166632.403644] buffer dirty: block 35498 count 3
[166632.407979] buffer dirty: block 35499 count 1
[166632.412221] buffer dirty: block 35501 count 2
[166632.416464] buffer dirty: block 35502 count 1
[166632.420726] buffer dirty: block 35503 count 1
[166632.424962] buffer dirty: block 35504 count 3
[166632.429169] buffer dirty: block 35505 count 1
[166632.433410] buffer dirty: block 35506 count 1
[166632.437580] buffer dirty: block 35507 count 3
[166632.441787] buffer dirty: block 35508 count 1
[166632.446056] buffer dirty: block 35509 count 2
[166632.450333] buffer dirty: block 35510 count 2
[166632.454579] buffer dirty: block 35512 count 1
[166632.458791] buffer dirty: block 35513 count 1
[166632.462964] buffer dirty: block 35514 count 2
[166632.467136] buffer dirty: block 35515 count 1
[166632.471309] buffer dirty: block 35517 count 1
[166632.475482] buffer dirty: block 35518 count 2
[166632.479651] buffer dirty: block 35519 count 2
[166632.483933] buffer dirty: block 35522 count 1
[166632.488066] buffer dirty: block 35523 count 1
[166632.492162] buffer dirty: block 35524 count 1
[166632.496258] buffer dirty: block 35525 count 4
[166632.500350] buffer dirty: block 35527 count 1
[166632.504476] buffer dirty: block 35528 count 2
[166632.508607] buffer dirty: block 35529 count 3
[166632.512736] buffer dirty: block 35533 count 3
[166632.516831] buffer dirty: block 35536 count 1
[166632.520973] buffer dirty: block 35537 count 3
[166632.525138] buffer dirty: block 35538 count 1
[166632.529267] buffer dirty: block 35539 count 3
[166632.533398] buffer dirty: block 35541 count 1
[166632.537526] buffer dirty: block 35542 count 2
[166632.541653] buffer dirty: block 35543 count 1
[166632.545820] buffer dirty: block 35544 count 1
[166632.549949] buffer dirty: block 35545 count 2
[166632.554048] buffer dirty: block 35547 count 3
[166632.558216] buffer dirty: block 35548 count 2
[166632.562388] buffer dirty: block 35549 count 1
[166632.566521] buffer dirty: block 35550 count 1
[166632.570624] buffer dirty: block 35552 count 1
[166632.574755] buffer dirty: block 35553 count 2
[166632.578887] buffer dirty: block 35554 count 1
[166632.583055] buffer dirty: block 35555 count 2
[166632.587226] buffer dirty: block 35556 count 1
[166632.591395] buffer dirty: block 35557 count 2
[166632.595574] buffer dirty: block 35558 count 1
[166632.599780] buffer dirty: block 35559 count 1
[166632.603989] buffer dirty: block 35560 count 3
[166632.608196] buffer dirty: block 35563 count 1
[166632.612446] buffer dirty: block 35564 count 3
[166632.616729] buffer dirty: block 35565 count 1
[166632.621013] buffer dirty: block 35567 count 1
[166632.625222] buffer dirty: block 35568 count 3
[166632.629432] buffer dirty: block 35569 count 1
[166632.633643] buffer dirty: block 35570 count 1
[166632.637849] buffer dirty: block 35571 count 1
[166632.642020] buffer dirty: block 35572 count 2
[166632.646190] buffer dirty: block 35573 count 2
[166632.650316] buffer dirty: block 35574 count 1
[166632.654448] buffer dirty: block 35575 count 2
[166632.658569] buffer dirty: block 35576 count 1
[166632.662693] buffer dirty: block 35577 count 1
[166632.666776] buffer dirty: block 35578 count 2
[166632.670902] buffer dirty: block 35579 count 2
[166632.674989] buffer dirty: block 35580 count 1
[166632.679078] buffer dirty: block 35583 count 2
[166632.683166] buffer dirty: block 35584 count 2
[166632.687216] buffer dirty: block 35585 count 1
[166632.691299] buffer dirty: block 35587 count 2
[166632.695420] buffer dirty: block 35588 count 1
[166632.699540] buffer dirty: block 35589 count 2
[166632.703657] buffer dirty: block 35590 count 1
[166632.707955] buffer dirty: block 35591 count 1
[166632.712075] buffer dirty: block 35592 count 1
[166632.716151] buffer dirty: block 35593 count 1
[166632.720187] buffer dirty: block 35594 count 3
[166632.724299] buffer dirty: block 35595 count 1
[166632.728375] buffer dirty: block 35597 count 2
[166632.732556] buffer dirty: block 35598 count 2
[166632.736610] buffer dirty: block 35599 count 2
[166632.740614] buffer dirty: block 35600 count 1
[166632.744688] buffer dirty: block 35603 count 4
[166632.748791] buffer dirty: block 35605 count 1
[166632.752897] buffer dirty: block 35606 count 1
[166632.757033] buffer dirty: block 35607 count 1
[166632.761136] buffer dirty: block 35608 count 2
[166632.765271] buffer dirty: block 35609 count 2
[166632.769409] buffer dirty: block 35610 count 1
[166632.773546] buffer dirty: block 35611 count 1
[166632.777652] buffer dirty: block 35613 count 2
[166632.781755] buffer dirty: block 35614 count 2
[166632.785898] buffer dirty: block 35616 count 2
[166632.790067] buffer dirty: block 35617 count 1
[166632.794205] buffer dirty: block 35618 count 3
[166632.798306] buffer dirty: block 35619 count 1
[166632.802439] buffer dirty: block 35620 count 1
[166632.806579] buffer dirty: block 35621 count 1
[166632.810682] buffer dirty: block 35622 count 2
[166632.814749] buffer dirty: block 35623 count 1
[166632.818817] buffer dirty: block 35624 count 1
[166632.822850] buffer dirty: block 35626 count 4
[166632.826910] buffer dirty: block 35628 count 2
[166632.830971] buffer dirty: block 35629 count 1
[166632.835112] buffer dirty: block 35630 count 2
[166632.839290] buffer dirty: block 35632 count 1
[166632.843435] buffer dirty: block 35633 count 1
[166632.847543] buffer dirty: block 35634 count 1
[166632.851611] buffer dirty: block 35635 count 1
[166632.855679] buffer dirty: block 35636 count 1
[166632.859782] buffer dirty: block 35637 count 1
[166632.863805] buffer dirty: block 35638 count 5
[166632.867796] buffer dirty: block 35639 count 1
[166632.871789] buffer dirty: block 35642 count 1
[166632.875784] buffer dirty: block 35643 count 2
[166632.879971] buffer dirty: block 35644 count 2
[166632.883960] buffer dirty: block 35645 count 1
[166632.888025] buffer dirty: block 35646 count 2
[166632.892087] buffer dirty: block 35648 count 3
[166632.896154] buffer dirty: block 35649 count 1
[166632.900291] buffer dirty: block 35650 count 2
[166632.904422] buffer dirty: block 35652 count 1
[166632.908574] buffer dirty: block 35653 count 2
[166632.912757] buffer dirty: block 35654 count 2
[166632.916904] buffer dirty: block 35656 count 3
[166632.921019] buffer dirty: block 35657 count 1
[166632.925166] buffer dirty: block 35658 count 1
[166632.929317] buffer dirty: block 35659 count 2
[166632.933461] buffer dirty: block 35660 count 1
[166632.937615] buffer dirty: block 35663 count 1
[166632.941768] buffer dirty: block 35664 count 2
[166632.945888] buffer dirty: block 35665 count 1
[166632.950007] buffer dirty: block 35666 count 1
[166632.954054] buffer dirty: block 35667 count 1
[166632.958098] buffer dirty: block 35668 count 1
[166632.962130] buffer dirty: block 35669 count 2
[166632.966199] buffer dirty: block 35670 count 3
[166632.970344] buffer dirty: block 35673 count 1
[166632.974530] buffer dirty: block 35674 count 1
[166632.978683] buffer dirty: block 35675 count 2
[166632.982876] buffer dirty: block 35676 count 1
[166632.987069] buffer dirty: block 35678 count 1
[166632.991219] buffer dirty: block 35679 count 4
[166632.995372] buffer dirty: block 35683 count 2
[166632.999496] buffer dirty: block 35684 count 1
[166633.003662] buffer dirty: block 35685 count 1
[166633.007809] buffer dirty: block 35686 count 1
[166633.011994] buffer dirty: block 35687 count 1
[166633.016019] buffer dirty: block 35688 count 2
[166633.020042] buffer dirty: block 35689 count 2
[166633.024052] buffer dirty: block 35690 count 1
[166633.028096] buffer dirty: block 35691 count 1
[166633.032132] buffer dirty: block 35693 count 1
[166633.036170] buffer dirty: block 35694 count 1
[166633.040205] buffer dirty: block 35696 count 2
[166633.044235] buffer dirty: block 35697 count 1
[166633.048338] buffer dirty: block 35698 count 1
[166633.052433] buffer dirty: block 35699 count 3
[166633.056548] buffer dirty: block 35701 count 1
[166633.060669] buffer dirty: block 35702 count 1
[166633.064800] buffer dirty: block 35703 count 2
[166633.068934] buffer dirty: block 35704 count 1
[166633.073105] buffer dirty: block 35705 count 1
[166633.077244] buffer dirty: block 35706 count 1
[166633.081350] buffer dirty: block 35707 count 1
[166633.085417] buffer dirty: block 35708 count 1
[166633.089409] buffer dirty: block 35709 count 2
[166633.093433] buffer dirty: block 35710 count 1
[166633.097534] buffer dirty: block 35711 count 1
[166633.101628] buffer dirty: block 35712 count 1
[166633.105686] buffer dirty: block 35713 count 2
[166633.109818] buffer dirty: block 35715 count 2
[166633.113908] buffer dirty: block 35716 count 2
[166633.117958] buffer dirty: block 35717 count 1
[166633.122037] buffer dirty: block 35718 count 2
[166633.126112] buffer dirty: block 35719 count 1
[166633.130229] buffer dirty: block 35720 count 1
[166633.134346] buffer dirty: block 35721 count 1
[166633.138425] buffer dirty: block 35722 count 2
[166633.142506] buffer dirty: block 35723 count 3
[166633.146558] buffer dirty: block 35726 count 2
[166633.150646] buffer dirty: block 35728 count 3
[166633.154777] buffer dirty: block 35729 count 1
[166633.158914] buffer dirty: block 35730 count 1
[166633.163011] buffer dirty: block 35731 count 1
[166633.167076] buffer dirty: block 35732 count 2
[166633.171179] buffer dirty: block 35733 count 2
[166633.175242] buffer dirty: block 35734 count 2
[166633.179260] buffer dirty: block 35735 count 1
[166633.183519] buffer dirty: block 35736 count 1
[166633.187544] buffer dirty: block 35738 count 4
[166633.191633] buffer dirty: block 35739 count 2
[166633.195897] buffer dirty: block 35741 count 1
[166633.200026] buffer dirty: block 35742 count 1
[166633.204191] buffer dirty: block 35743 count 2
[166633.208362] buffer dirty: block 35746 count 2
[166633.212499] buffer dirty: block 35748 count 3
[166633.216599] buffer dirty: block 35749 count 2
[166633.220699] buffer dirty: block 35750 count 1
[166633.224833] buffer dirty: block 35752 count 2
[166633.228931] buffer dirty: block 35753 count 3
[166633.233111] buffer dirty: block 35754 count 1
[166633.237313] buffer dirty: block 35756 count 1
[166633.241488] buffer dirty: block 35758 count 4
[166633.245667] buffer dirty: block 35759 count 2
[166633.249841] buffer dirty: block 35760 count 1
[166633.254052] buffer dirty: block 35762 count 2
[166633.258270] buffer dirty: block 35763 count 3
[166633.262523] buffer dirty: block 35764 count 1
[166633.266776] buffer dirty: block 35765 count 1
[166633.270959] buffer dirty: block 35766 count 2
[166633.275140] buffer dirty: block 35768 count 2
[166633.279283] buffer dirty: block 35769 count 2
[166633.283421] buffer dirty: block 35771 count 2
[166633.287529] buffer dirty: block 35772 count 1
[166633.291639] buffer dirty: block 35773 count 1
[166633.295712] buffer dirty: block 35775 count 1
[166633.299970] buffer dirty: block 35776 count 1
[166633.303976] buffer dirty: block 35777 count 2
[166633.308047] buffer dirty: block 35778 count 1
[166633.312089] buffer dirty: block 35779 count 1
[166633.316122] buffer dirty: block 35780 count 1
[166633.320119] buffer dirty: block 35781 count 2
[166633.324185] buffer dirty: block 35782 count 2
[166633.328244] buffer dirty: block 35784 count 1
[166633.332323] buffer dirty: block 35785 count 1
[166633.336382] buffer dirty: block 35786 count 1
[166633.340441] buffer dirty: block 35787 count 2
[166633.344503] buffer dirty: block 35788 count 1
[166633.348564] buffer dirty: block 35789 count 2
[166633.352625] buffer dirty: block 35792 count 5
[166633.356694] buffer dirty: block 35793 count 1
[166633.360796] buffer dirty: block 35795 count 1
[166633.364898] buffer dirty: block 35796 count 2
[166633.369004] buffer dirty: block 35797 count 1
[166633.373136] buffer dirty: block 35798 count 4
[166633.377268] buffer dirty: block 35799 count 2
[166633.381398] buffer dirty: block 35800 count 1
[166633.385531] buffer dirty: block 35801 count 1
[166633.389664] buffer dirty: block 35802 count 1
[166633.393754] buffer dirty: block 35803 count 1
[166633.397846] buffer dirty: block 35804 count 1
[166633.401937] buffer dirty: block 35806 count 1
[166633.406030] buffer dirty: block 35807 count 1
[166633.410083] buffer dirty: block 35808 count 2
[166633.414210] buffer dirty: block 35809 count 2
[166633.418341] buffer dirty: block 35811 count 1
[166633.422479] buffer dirty: block 35812 count 1
[166633.426576] buffer dirty: block 35813 count 1
[166633.430668] buffer dirty: block 35815 count 2
[166633.434769] buffer dirty: block 35816 count 4
[166633.438936] buffer dirty: block 35818 count 1
[166633.443067] buffer dirty: block 35819 count 2
[166633.447205] buffer dirty: block 35820 count 1
[166633.451348] buffer dirty: block 35821 count 1
[166633.455492] buffer dirty: block 35822 count 1
[166633.459603] buffer dirty: block 35824 count 2
[166633.463711] buffer dirty: block 35825 count 1
[166633.467821] buffer dirty: block 35826 count 2
[166633.471998] buffer dirty: block 35827 count 2
[166633.476031] buffer dirty: block 35828 count 2
[166633.479995] buffer dirty: block 35829 count 1
[166633.483957] buffer dirty: block 35830 count 2
[166633.484035] buffer dirty: block 35832 count 2
[166633.488006] buffer dirty: block 35834 count 2
[166633.491937] buffer dirty: block 35835 count 2
[166633.495863] buffer dirty: block 35836 count 2
[166633.499748] buffer dirty: block 35838 count 2
[166633.503592] buffer dirty: block 35839 count 1
[166633.507479] buffer dirty: block 35840 count 2
[166633.511407] buffer dirty: block 35842 count 2
[166633.515374] buffer dirty: block 35843 count 1
[166633.519376] buffer dirty: block 35844 count 1
[166633.523369] buffer dirty: block 35845 count 2
[166633.527366] buffer dirty: block 35846 count 2
[166633.531393] buffer dirty: block 35848 count 2
[166633.535411] buffer dirty: block 35849 count 2
[166633.539425] buffer dirty: block 35851 count 1
[166633.543484] buffer dirty: block 35852 count 3
[166633.547574] buffer dirty: block 35853 count 1
[166633.551630] buffer dirty: block 35854 count 1
[166633.555645] buffer dirty: block 35856 count 2
[166633.559692] buffer dirty: block 35857 count 1
[166633.563768] buffer dirty: block 35858 count 1
[166633.567843] buffer dirty: block 35859 count 3
[166633.571960] buffer dirty: block 35861 count 1
[166633.576081] buffer dirty: block 35862 count 4
[166633.583931] buffer dirty: block 35866 count 1
[166633.588061] buffer dirty: block 35867 count 1
[166633.592174] buffer dirty: block 35868 count 2
[166633.596299] buffer dirty: block 35869 count 2
[166633.600418] buffer dirty: block 35872 count 2
[166633.604467] buffer dirty: block 35874 count 1
[166633.608587] buffer dirty: block 35876 count 1
[166633.612672] buffer dirty: block 35877 count 3
[166633.616798] buffer dirty: block 35878 count 2
[166633.620961] buffer dirty: block 35879 count 1
[166633.625124] buffer dirty: block 35880 count 2
[166633.629328] buffer dirty: block 35881 count 2
[166633.633523] buffer dirty: block 35882 count 2
[166633.637897] buffer dirty: block 1081370 count 1
[166633.642089] buffer dirty: block 2588673 count 1
[166633.646391] buffer dirty: block 2588727 count 2
[166633.650767] Total number of dirty buffers: 24754
for /home
and
[166982.440144] buffer dirty: block 31549 count 1
[166982.443561] buffer dirty: block 31573 count 1
[166982.446978] buffer dirty: block 31586 count 1
[166982.450372] buffer dirty: block 31590 count 1
[166982.453800] buffer dirty: block 31601 count 1
[166982.457235] buffer dirty: block 31609 count 1
[166982.460667] buffer dirty: block 31611 count 1
[166982.464047] buffer dirty: block 31614 count 1
[166982.467477] buffer dirty: block 31615 count 1
[166982.470903] buffer dirty: block 31621 count 1
[166982.474333] buffer dirty: block 31634 count 1
[166982.477760] buffer dirty: block 31644 count 1
[166982.481154] buffer dirty: block 31645 count 1
[166982.484538] buffer dirty: block 31648 count 2
[166982.487915] buffer dirty: block 31651 count 1
[166982.491929] buffer dirty: block 31659 count 1
[166982.495424] buffer dirty: block 31660 count 1
[166982.498919] buffer dirty: block 31662 count 1
[166982.502421] buffer dirty: block 31666 count 1
[166982.505912] buffer dirty: block 31667 count 1
[166982.509365] buffer dirty: block 31668 count 1
[166982.512218] buffer dirty: block 31676 count 1
[166982.515629] buffer dirty: block 31697 count 1
[166982.519043] buffer dirty: block 31700 count 1
[166982.522461] buffer dirty: block 31701 count 1
[166982.525872] buffer dirty: block 31704 count 1
[166982.529258] buffer dirty: block 31708 count 1
[166982.532642] buffer dirty: block 31713 count 1
[166982.536013] buffer dirty: block 31738 count 2
[166982.539441] buffer dirty: block 31751 count 2
[166982.542868] buffer dirty: block 31754 count 1
[166982.546336] buffer dirty: block 31764 count 1
[166982.549799] buffer dirty: block 31787 count 1
[166982.553225] buffer dirty: block 31788 count 1
[166982.556649] buffer dirty: block 31795 count 1
[166982.560026] buffer dirty: block 31796 count 1
[166982.563448] buffer dirty: block 31809 count 1
[166982.566881] buffer dirty: block 31810 count 1
[166982.570324] buffer dirty: block 31822 count 1
[166982.573766] buffer dirty: block 31833 count 1
[166982.577208] buffer dirty: block 31834 count 1
[166982.580648] buffer dirty: block 31837 count 1
[166982.584060] buffer dirty: block 31840 count 1
[166982.587460] buffer dirty: block 31843 count 1
[166982.590891] buffer dirty: block 31846 count 1
[166982.594249] buffer dirty: block 31849 count 1
[166982.597571] buffer dirty: block 31862 count 1
[166982.600859] buffer dirty: block 31878 count 1
[166982.604131] buffer dirty: block 31886 count 1
[166982.607427] buffer dirty: block 31889 count 1
[166982.610722] buffer dirty: block 31896 count 1
[166982.614052] buffer dirty: block 31899 count 1
[166982.617373] buffer dirty: block 31903 count 1
[166982.620660] buffer dirty: block 31906 count 1
[166982.623963] buffer dirty: block 31907 count 1
[166982.627985] buffer dirty: block 31912 count 1
[166982.631376] buffer dirty: block 31920 count 1
[166982.634768] buffer dirty: block 31936 count 1
[166982.638119] buffer dirty: block 31937 count 1
[166982.641467] buffer dirty: block 31943 count 1
[166982.644111] buffer dirty: block 31947 count 1
[166982.647427] buffer dirty: block 31950 count 1
[166982.650742] buffer dirty: block 31953 count 1
[166982.654054] buffer dirty: block 31956 count 2
[166982.657401] buffer dirty: block 31967 count 1
[166982.660791] buffer dirty: block 31980 count 1
[166982.664088] buffer dirty: block 31981 count 1
[166982.667466] buffer dirty: block 31990 count 1
[166982.670845] buffer dirty: block 31998 count 1
[166982.674225] buffer dirty: block 31999 count 1
[166982.677602] buffer dirty: block 32002 count 1
[166982.680986] buffer dirty: block 32006 count 1
[166982.684363] buffer dirty: block 32030 count 1
[166982.687736] buffer dirty: block 32031 count 1
[166982.691751] buffer dirty: block 32032 count 1
[166982.695184] buffer dirty: block 32035 count 1
[166982.698611] buffer dirty: block 32049 count 1
[166982.702032] buffer dirty: block 32059 count 1
[166982.705452] buffer dirty: block 32065 count 1
[166982.708201] buffer dirty: block 32079 count 1
[166982.711583] buffer dirty: block 32082 count 1
[166982.714999] buffer dirty: block 32083 count 1
[166982.718395] buffer dirty: block 32096 count 1
[166982.721814] buffer dirty: block 32109 count 1
[166982.725212] buffer dirty: block 32123 count 1
[166982.728601] buffer dirty: block 32126 count 1
[166982.731973] buffer dirty: block 32132 count 1
[166982.736004] buffer dirty: block 32138 count 1
[166982.739399] buffer dirty: block 32146 count 1
[166982.742793] buffer dirty: block 32155 count 1
[166982.746225] buffer dirty: block 32156 count 1
[166982.749652] buffer dirty: block 32168 count 1
[166982.752417] buffer dirty: block 32169 count 1
[166982.755789] buffer dirty: block 32170 count 1
[166982.759844] buffer dirty: block 32171 count 1
[166982.763194] buffer dirty: block 32175 count 1
[166982.766542] buffer dirty: block 32181 count 1
[166982.769890] buffer dirty: block 32182 count 1
[166982.772614] buffer dirty: block 32184 count 1
[166982.775947] buffer dirty: block 32188 count 1
[166982.779961] buffer dirty: block 32189 count 1
[166982.783334] buffer dirty: block 32190 count 1
[166982.786711] buffer dirty: block 32191 count 1
[166982.790093] buffer dirty: block 32193 count 1
[166982.793437] buffer dirty: block 32194 count 1
[166982.796097] buffer dirty: block 32197 count 1
[166982.799432] buffer dirty: block 32198 count 1
[166982.802773] buffer dirty: block 32200 count 1
[166982.806115] buffer dirty: block 32201 count 1
[166982.809455] buffer dirty: block 32203 count 1
[166982.812788] buffer dirty: block 32208 count 1
[166982.816101] buffer dirty: block 32213 count 1
[166982.819398] buffer dirty: block 32220 count 1
[166982.822729] buffer dirty: block 32225 count 1
[166982.826023] buffer dirty: block 32230 count 2
[166982.829358] buffer dirty: block 32231 count 1
[166982.832620] buffer dirty: block 32234 count 1
[166982.835977] buffer dirty: block 32235 count 1
[166982.839992] buffer dirty: block 32239 count 1
[166982.843356] buffer dirty: block 32240 count 1
[166982.846727] buffer dirty: block 32243 count 1
[166982.850094] buffer dirty: block 32254 count 1
[166982.853461] buffer dirty: block 32273 count 1
[166982.856173] buffer dirty: block 32274 count 1
[166982.859537] buffer dirty: block 32280 count 1
[166982.862909] buffer dirty: block 32314 count 1
[166982.866287] buffer dirty: block 32322 count 1
[166982.869667] buffer dirty: block 32323 count 1
[166982.873043] buffer dirty: block 32326 count 1
[166982.876407] buffer dirty: block 32329 count 1
[166982.879733] buffer dirty: block 32332 count 1
[166982.883789] buffer dirty: block 32338 count 1
[166982.887166] buffer dirty: block 32345 count 1
[166982.890544] buffer dirty: block 32346 count 1
[166982.893925] buffer dirty: block 32356 count 1
[166982.896590] buffer dirty: block 32357 count 1
[166982.899956] buffer dirty: block 32361 count 1
[166982.903980] buffer dirty: block 32362 count 1
[166982.907394] buffer dirty: block 32365 count 1
[166982.910813] buffer dirty: block 32368 count 1
[166982.914233] buffer dirty: block 32371 count 1
[166982.917648] buffer dirty: block 32374 count 1
[166982.920385] buffer dirty: block 32381 count 1
[166982.923781] buffer dirty: block 32396 count 1
[166982.927797] buffer dirty: block 32405 count 1
[166982.931177] buffer dirty: block 32413 count 1
[166982.934555] buffer dirty: block 32421 count 2
[166982.937974] buffer dirty: block 32422 count 1
[166982.941399] buffer dirty: block 32425 count 1
[166982.944154] buffer dirty: block 32428 count 1
[166982.947571] buffer dirty: block 32432 count 1
[166982.950988] buffer dirty: block 32435 count 2
[166982.954443] buffer dirty: block 32436 count 1
[166982.957940] buffer dirty: block 32438 count 1
[166982.961467] buffer dirty: block 32446 count 2
[166982.965030] buffer dirty: block 32475 count 1
[166982.968630] buffer dirty: block 32478 count 2
[166982.972036] buffer dirty: block 32481 count 2
[166982.975707] buffer dirty: block 32482 count 1
[166982.979419] buffer dirty: block 32490 count 1
[166982.983130] buffer dirty: block 32491 count 1
[166982.986808] buffer dirty: block 32492 count 1
[166982.990482] buffer dirty: block 32514 count 1
[166982.994162] buffer dirty: block 32515 count 1
[166982.997839] buffer dirty: block 32522 count 1
[166983.001536] buffer dirty: block 32523 count 1
[166983.005292] buffer dirty: block 32526 count 1
[166983.009010] buffer dirty: block 32555 count 1
[166983.012729] buffer dirty: block 32556 count 1
[166983.016458] buffer dirty: block 32560 count 1
[166983.020089] buffer dirty: block 32563 count 1
[166983.023796] buffer dirty: block 32566 count 1
[166983.027825] buffer dirty: block 32569 count 1
[166983.031508] buffer dirty: block 32584 count 1
[166983.035157] buffer dirty: block 32593 count 1
[166983.038806] buffer dirty: block 32660 count 1
[166983.042422] buffer dirty: block 32666 count 1
[166983.046000] buffer dirty: block 32680 count 1
[166983.049579] buffer dirty: block 32703 count 1
[166983.053165] buffer dirty: block 32706 count 1
[166983.056750] buffer dirty: block 32709 count 1
[166983.060163] buffer dirty: block 32716 count 1
[166983.063670] buffer dirty: block 32719 count 1
[166983.067219] buffer dirty: block 32722 count 1
[166983.070689] buffer dirty: block 32723 count 1
[166983.074134] buffer dirty: block 32726 count 1
[166983.077526] buffer dirty: block 32729 count 1
[166983.080888] buffer dirty: block 32744 count 1
[166983.084157] buffer dirty: block 32767 count 1
[166983.087458] buffer dirty: block 33329 count 1
[166983.090791] buffer dirty: block 33337 count 2
[166983.094195] buffer dirty: block 33708 count 1
[166983.097642] buffer dirty: block 33711 count 1
[166983.101091] buffer dirty: block 33726 count 1
[166983.104412] buffer dirty: block 33735 count 1
[166983.107774] buffer dirty: block 33755 count 1
[166983.111869] buffer dirty: block 33789 count 1
[166983.115252] buffer dirty: block 33797 count 1
[166983.118638] buffer dirty: block 33798 count 1
[166983.122017] buffer dirty: block 33811 count 1
[166983.125405] buffer dirty: block 33835 count 1
[166983.128131] buffer dirty: block 33836 count 1
[166983.131518] buffer dirty: block 33839 count 1
[166983.134911] buffer dirty: block 33843 count 1
[166983.138330] buffer dirty: block 33847 count 1
[166983.141742] buffer dirty: block 33848 count 1
[166983.145156] buffer dirty: block 33851 count 1
[166983.148544] buffer dirty: block 33865 count 1
[166983.151935] buffer dirty: block 33888 count 1
[166983.155959] buffer dirty: block 33891 count 1
[166983.159407] buffer dirty: block 33899 count 1
[166983.162849] buffer dirty: block 33902 count 1
[166983.166302] buffer dirty: block 33903 count 1
[166983.169750] buffer dirty: block 33933 count 1
[166983.172554] buffer dirty: block 33970 count 1
[166983.175986] buffer dirty: block 34007 count 1
[166983.180001] buffer dirty: block 34044 count 1
[166983.183500] buffer dirty: block 34081 count 1
[166983.187029] buffer dirty: block 34118 count 1
[166983.190525] buffer dirty: block 34127 count 1
[166983.194015] buffer dirty: block 34140 count 1
[166983.197426] buffer dirty: block 34164 count 1
[166983.200403] buffer dirty: block 34167 count 1
[166983.203799] buffer dirty: block 34168 count 1
[166983.207813] buffer dirty: block 34171 count 1
[166983.211258] buffer dirty: block 34174 count 1
[166983.214695] buffer dirty: block 34177 count 1
[166983.218129] buffer dirty: block 34180 count 1
[166983.221565] buffer dirty: block 34214 count 1
[166983.224433] buffer dirty: block 34222 count 1
[166983.227847] buffer dirty: block 34223 count 1
[166983.231864] buffer dirty: block 34226 count 1
[166983.235337] buffer dirty: block 34239 count 1
[166983.238772] buffer dirty: block 34263 count 1
[166983.242213] buffer dirty: block 34264 count 1
[166983.245656] buffer dirty: block 34267 count 1
[166983.248520] buffer dirty: block 34283 count 1
[166983.251940] buffer dirty: block 34291 count 1
[166983.255959] buffer dirty: block 34358 count 1
[166983.259397] buffer dirty: block 34364 count 1
[166983.262837] buffer dirty: block 34378 count 1
[166983.266273] buffer dirty: block 34401 count 1
[166983.269711] buffer dirty: block 34407 count 1
[166983.272572] buffer dirty: block 34410 count 1
[166983.275959] buffer dirty: block 34413 count 1
[166983.280004] buffer dirty: block 34421 count 1
[166983.283404] buffer dirty: block 34424 count 1
[166983.286759] buffer dirty: block 34425 count 1
[166983.290109] buffer dirty: block 34428 count 1
[166983.293420] buffer dirty: block 34431 count 1
[166983.296198] buffer dirty: block 34432 count 1
[166983.299505] buffer dirty: block 34461 count 1
[166983.302814] buffer dirty: block 34462 count 1
[166983.306093] buffer dirty: block 34468 count 1
[166983.309370] buffer dirty: block 34484 count 1
[166983.312644] buffer dirty: block 34492 count 1
[166983.315909] buffer dirty: block 34513 count 1
[166983.319930] buffer dirty: block 34547 count 1
[166983.323247] buffer dirty: block 34555 count 1
[166983.326563] buffer dirty: block 34556 count 1
[166983.329879] buffer dirty: block 34560 count 1
[166983.332417] buffer dirty: block 34564 count 1
[166983.335697] buffer dirty: block 34567 count 1
[166983.338974] buffer dirty: block 34570 count 1
[166983.342253] buffer dirty: block 34583 count 1
[166983.345535] buffer dirty: block 34607 count 1
[166983.348823] buffer dirty: block 34608 count 1
[166983.352086] buffer dirty: block 34611 count 1
[166983.355332] buffer dirty: block 34614 count 1
[166983.358613] buffer dirty: block 34627 count 1
[166983.361863] buffer dirty: block 34651 count 1
[166983.365115] buffer dirty: block 34654 count 1
[166983.368339] buffer dirty: block 34662 count 1
[166983.371578] buffer dirty: block 34665 count 1
[166983.374825] buffer dirty: block 34668 count 1
[166983.378073] buffer dirty: block 34671 count 1
[166983.381390] buffer dirty: block 131150 count 1
[166983.384564] buffer dirty: block 131151 count 2
[166983.387911] buffer dirty: block 131152 count 1
[166983.391926] buffer dirty: block 131153 count 2
[166983.395369] buffer dirty: block 131154 count 1
[166983.398847] buffer dirty: block 131155 count 2
[166983.402359] buffer dirty: block 131156 count 1
[166983.405914] buffer dirty: block 131157 count 1
[166983.409471] buffer dirty: block 131159 count 1
[166983.412215] buffer dirty: block 131189 count 2
[166983.415897] buffer dirty: block 491704 count 1
[166983.419541] buffer dirty: block 491929 count 1
[166983.423176] buffer dirty: block 491932 count 1
[166983.426839] buffer dirty: block 491933 count 1
[166983.430482] buffer dirty: block 524297 count 1
[166983.434135] buffer dirty: block 524301 count 1
[166983.437797] buffer dirty: block 524303 count 1
[166983.441455] buffer dirty: block 524304 count 2
[166983.445149] buffer dirty: block 524305 count 1
[166983.448881] buffer dirty: block 524310 count 1
[166983.452613] buffer dirty: block 524311 count 1
[166983.456164] buffer dirty: block 524315 count 1
[166983.459876] buffer dirty: block 524326 count 1
[166983.463895] buffer dirty: block 524331 count 1
[166983.467661] buffer dirty: block 524391 count 1
[166983.471583] buffer dirty: block 1114244 count 1
[166983.475404] buffer dirty: block 1114246 count 1
[166983.479290] buffer dirty: block 1212425 count 1
[166983.483192] buffer dirty: block 1245295 count 1
[166983.487102] buffer dirty: block 1245383 count 1
[166983.491036] buffer dirty: block 1278016 count 1
[166983.495167] Total number of dirty buffers: 5421
for /
On Sun, Oct 05, 2008 at 11:15:26AM +0200, Quentin Godfroy wrote:
> On Thu, Oct 02, 2008 at 08:35:48PM -0400, Theodore Tso wrote:
> > On Wed, Oct 01, 2008 at 12:23:58AM +0200, Quentin wrote:
> > >
> > > Of course. However since I unmounted and remounted /home the 'buffer' line
> > > is now only 59megs, and they are still not dropped when a program tries to
> > > malloc all the memory. I'll tell next time the problem shows up (it
> > > can take ten days)
> > >
> >
> > Are you willing to patch and recompile your kernel? If so, the
> > following patch would be very helpful in determining what is going on.
> > It allows us to see what buffer heads are in use for a particular
> > block device. Attached please the kernel patch and the user program.
>
> Now that the machine is again in the 100M+ in buffers (still unreleased when
> a program asks for all the memory), I launched the program on the devices
> which support / and /home.
>
> I also attached /proc/meminfo and /proc/slabinfo
>
> In both cases it freezes solid the machine for more than a minute or so, and
> it overflows the dmesg with messages.
Can you check and see if you got more of the messages recorded in
/var/log/messages? Once you do, can you take the block numbers, and
pull them out into a single command file to feed to debugfs.
So for example, given:
> [166632.382632] buffer dirty: block 35491 count 1
> [166632.386827] buffer dirty: block 35493 count 3
> [166632.391019] buffer dirty: block 35494 count 1
> [166632.395251] buffer dirty: block 35496 count 1
> [166632.399446] buffer dirty: block 35497 count 2
> [166632.403644] buffer dirty: block 35498 count 3
> [166632.407979] buffer dirty: block 35499 count 1
> [166632.412221] buffer dirty: block 35501 count 2
Take the column of block numbers, and tack on "icheck " at the
beginning, like so:
icheck 35491 35493 35494 35496 35497 35498 35499 35501 ...
You can put a thousand or so block numbers on each line; then it's
probably better to start a new line with "icheck " at the beginning.
Then take that script and run it through debugfs:
debugfs /dev/XXX < icheck.in > icheck.out
That will result in a file icheck.out that looks like this:
debugfs: icheck 33347
Block Inode number
33347 8193
33348 8193
33350 8196
33351 8197
...
Now you'll need to take the inode numbers returned in icheck.out, and
create another file called ncheck.in that will take the inode numbers
and turn them into pathnames. (I find that using emacs's
kill-rectangle command very handy for doing this sort of thing, but
other people will like to use awk, and I'm sure there's some way to do
this using vi but I don't know what it is. :-) It's also a good idea
to take the inode numbers and run them through "sort -u" to get rid of
duplicates before putting them on a single line and adding ncheck to
them. So what you want is to create a file ncheck.in that looks like this:
ncheck 8193 8196 8197 ....
... and then feed that to debugfs again:
debugfs /dev/XXX < ncheck.in > ncheck.out
That will produce a file that looks like this:
debugfs: ncheck 8193
Inode Pathname
8193 /ext4
...
The next thing I'd ask you to do is to look at the pathnames and
eyeball them; are they all directories? Files? Files that you have
modified earlier? If you're not sure, you can look at a particular
inode either by giving its pathname:
debugfs: stat /ext4
or by its inode number, in angle brackets:
debugfs: stat <8193>
What I'm trying to do here is to get a pattern of what might be going
on. I'm assuming that your filesystem is too big (and probably
contains private information) for you to send it to me. (Although if
you're willing to send me a compressed raw e2image --- see the "RAW
IMAGE FILES" section of the e2image man page ---- and the portions of
the buffer information dummped in /var/log/messages, I can try to do
some of the analysis for you.)
- Ted
On Sun, Oct 05, 2008 at 08:27:52AM -0400, Theodore Tso wrote:
> > In both cases it freezes solid the machine for more than a minute or so, and
> > it overflows the dmesg with messages.
>
> Can you check and see if you got more of the messages recorded in
> /var/log/messages? Once you do, can you take the block numbers, and
> pull them out into a single command file to feed to debugfs.
Unfortunately, there are no more messages in the syslog than in the dmesg.
...
> Take the column of block numbers, and tack on "icheck " at the
> beginning, like so:
>
> icheck 35491 35493 35494 35496 35497 35498 35499 35501 ...
>
> You can put a thousand or so block numbers on each line; then it's
> probably better to start a new line with "icheck " at the beginning.
> Then take that script and run it through debugfs:
>
> debugfs /dev/XXX < icheck.in > icheck.out
>
> That will result in a file icheck.out that looks like this:
>
> debugfs: icheck 33347
> Block Inode number
> 33347 8193
> 33348 8193
> 33350 8196
> 33351 8197
> ...
>
> Now you'll need to take the inode numbers returned in icheck.out, and
> create another file called ncheck.in that will take the inode numbers
> and turn them into pathnames. (I find that using emacs's
> kill-rectangle command very handy for doing this sort of thing, but
> other people will like to use awk, and I'm sure there's some way to do
> this using vi but I don't know what it is. :-) It's also a good idea
> to take the inode numbers and run them through "sort -u" to get rid of
> duplicates before putting them on a single line and adding ncheck to
> them. So what you want is to create a file ncheck.in that looks like this:
>
> ncheck 8193 8196 8197 ....
>
> ... and then feed that to debugfs again:
>
> debugfs /dev/XXX < ncheck.in > ncheck.out
>
> That will produce a file that looks like this:
>
> debugfs: ncheck 8193
> Inode Pathname
> 8193 /ext4
> ...
>
>
> The next thing I'd ask you to do is to look at the pathnames and
> eyeball them; are they all directories? Files? Files that you have
> modified earlier? If you're not sure, you can look at a particular
> inode either by giving its pathname:
>
> debugfs: stat /ext4
>
> or by its inode number, in angle brackets:
>
> debugfs: stat <8193>
>
> What I'm trying to do here is to get a pattern of what might be going
> on. I'm assuming that your filesystem is too big (and probably
> contains private information) for you to send it to me. (Although if
> you're willing to send me a compressed raw e2image --- see the "RAW
> IMAGE FILES" section of the e2image man page ---- and the portions of
> the buffer information dummped in /var/log/messages, I can try to do
> some of the analysis for you.)
For the two fs the only inode which shows up is the inode 8 (this seems to
be the journal. According to 'stat <8>' in debugfs it looks like the journal is 134Megs long. I don't remember exactly how I created the fs, but i'm sure I did not specified the journal size. Does it seem reasonable for a 6,6G fs?
However seeing only the journal inode could be because I only had the last
300 blocks left in the dmesg. A small number of them did not belong to any
inode.
Should I reboot with a very big ring buffer?
On Sun, Oct 05, 2008 at 06:12:15PM +0200, Quentin Godfroy wrote:
> For the two fs the only inode which shows up is the inode 8 (this
> seems to be the journal. According to 'stat <8>' in debugfs it looks
> like the journal is 134Megs long. I don't remember exactly how I
> created the fs, but i'm sure I did not specified the journal
> size. Does it seem reasonable for a 6,6G fs?
134 Megs sounds wrong. What does dumpe2fs -h say? I'm guessing you
didn't calculate it quite correctly.
I did some poking around myself, and noticed that a lot of in-use
buffers hanging around from the journal inode. The following patch
should fix that problem. I'm still doing some more testingto make
sure there aren't any other buffer head leaks, but this is seems to
fix the worst of the problems. Can you let me know how this works for
you?
- Ted
jbd2: Fix buffer head leak when writing the commit block
Also make sure the buffer heads are marked clean before submitting bh
for writing. The previous code was marking the buffer head dirty,
which would have forced an unneeded write (and seek) to the journal
for no good reason.
Signed-off-by: "Theodore Ts'o" <[email protected]>
diff --git a/fs/jbd2/commit.c b/fs/jbd2/commit.c
index e91f051..c2b04cd 100644
--- a/fs/jbd2/commit.c
+++ b/fs/jbd2/commit.c
@@ -127,8 +127,7 @@ static int journal_submit_commit_record(journal_t *journal,
JBUFFER_TRACE(descriptor, "submit commit block");
lock_buffer(bh);
- get_bh(bh);
- set_buffer_dirty(bh);
+ clear_buffer_dirty(bh);
set_buffer_uptodate(bh);
bh->b_end_io = journal_end_buffer_io_sync;
@@ -157,7 +156,7 @@ static int journal_submit_commit_record(journal_t *journal,
/* And try again, without the barrier */
lock_buffer(bh);
- set_buffer_uptodate(bh);
+ clear_buffer_uptodate(bh);
set_buffer_dirty(bh);
ret = submit_bh(WRITE, bh);
}
Theodore Tso wrote:
> On Sun, Oct 05, 2008 at 06:12:15PM +0200, Quentin Godfroy wrote:
>> For the two fs the only inode which shows up is the inode 8 (this
>> seems to be the journal. According to 'stat <8>' in debugfs it looks
>> like the journal is 134Megs long. I don't remember exactly how I
>> created the fs, but i'm sure I did not specified the journal
>> size. Does it seem reasonable for a 6,6G fs?
>
> 134 Megs sounds wrong. What does dumpe2fs -h say? I'm guessing you
> didn't calculate it quite correctly.
>
> I did some poking around myself, and noticed that a lot of in-use
> buffers hanging around from the journal inode. The following patch
> should fix that problem. I'm still doing some more testingto make
> sure there aren't any other buffer head leaks, but this is seems to
> fix the worst of the problems. Can you let me know how this works for
> you?
>
> - Ted
>
> jbd2: Fix buffer head leak when writing the commit block
>
> Also make sure the buffer heads are marked clean before submitting bh
> for writing. The previous code was marking the buffer head dirty,
> which would have forced an unneeded write (and seek) to the journal
> for no good reason.
>
> Signed-off-by: "Theodore Ts'o" <[email protected]>
> diff --git a/fs/jbd2/commit.c b/fs/jbd2/commit.c
> index e91f051..c2b04cd 100644
> --- a/fs/jbd2/commit.c
> +++ b/fs/jbd2/commit.c
> @@ -127,8 +127,7 @@ static int journal_submit_commit_record(journal_t *journal,
>
> JBUFFER_TRACE(descriptor, "submit commit block");
> lock_buffer(bh);
> - get_bh(bh);
> - set_buffer_dirty(bh);
> + clear_buffer_dirty(bh);
> set_buffer_uptodate(bh);
> bh->b_end_io = journal_end_buffer_io_sync;
>
> @@ -157,7 +156,7 @@ static int journal_submit_commit_record(journal_t *journal,
>
> /* And try again, without the barrier */
> lock_buffer(bh);
> - set_buffer_uptodate(bh);
> + clear_buffer_uptodate(bh);
> set_buffer_dirty(bh);
> ret = submit_bh(WRITE, bh);
> }
Just so it doesn't get lost (discussed w/ Ted today) I think this 2nd
hunk flipped the wrong buffer funtion; this makes much more sense to me:
@@ -157,7 +156,7 @@ static int journal_submit_commit_record(journal_t
*journal,
/* And try again, without the barrier */
lock_buffer(bh);
set_buffer_uptodate(bh);
- set_buffer_dirty(bh);
+ clear_buffer_dirty(bh);
ret = submit_bh(WRITE, bh);
}
-Eric
Theodore Tso wrote:
> Quentin Godfroy wrote:
>> For the two fs the only inode which shows up is the inode 8 (this
>> seems to be the journal. According to 'stat <8>' in debugfs it looks
>> like the journal is 134Megs long. I don't remember exactly how I
>> created the fs, but i'm sure I did not specified the journal
>> size. Does it seem reasonable for a 6,6G fs?
>
> 134 Megs sounds wrong. What does dumpe2fs -h say? I'm guessing you
> didn't calculate it quite correctly.
I'm running 2.6.26.2 (ext3) and dumpe2fs show me:
Journal size: 128M
# fdisk -l
Disk /dev/sda: 100.0 GB, 100030242816 bytes
255 heads, 63 sectors/track, 12161 cylinders
Units = cilindros of 16065 * 512 = 8225280 bytes
Disk identifier: 0xe5d80ea3
Dispositivo Boot Start End Blocks Id System
/dev/sda1 1 889 7139328 27 Unknown
Partition 1 does not end on cylinder boundary.
/dev/sda2 * 889 4780 31250000 7 HPFS ou NTFS
Partition 2 does not end on cylinder boundary.
/dev/sda3 4781 7212 19527480 83 Linux
Partition 3 does not end on cylinder boundary.
/dev/sda4 7212 12161 39758040 5 Extended
Partition 4 does not end on cylinder boundary.
/dev/sda5 7212 7274 498928+ 82 Linux swap
/dev/sda6 7274 12161 39259048+ 83 Linux
Regards,
Renato
Here are updates to the two patches I sent you. The first fixes some
printk labels and requires that the debugging dump ioctl be triggered
by root; the second fixes the think-o that Eric pointed out earlier
this morning.
- Ted
On Mon, Oct 06, 2008 at 01:55:02PM -0400, Theodore Tso wrote:
> Here are updates to the two patches I sent you. The first fixes some
> printk labels and requires that the debugging dump ioctl be triggered
> by root; the second fixes the think-o that Eric pointed out earlier
> this morning.
Hey Quentin, just wanted to check in and find out if the patches seem
to be working for you. They seem to solve the problem on my side; I
can no longer reproduce the problem. Is it working for you?
Thanks, regards,
- Ted
On Tue, Oct 07, 2008 at 06:12:56PM -0400, Theodore Tso wrote:
> On Mon, Oct 06, 2008 at 01:55:02PM -0400, Theodore Tso wrote:
> > Here are updates to the two patches I sent you. The first fixes some
> > printk labels and requires that the debugging dump ioctl be triggered
> > by root; the second fixes the think-o that Eric pointed out earlier
> > this morning.
>
> Hey Quentin, just wanted to check in and find out if the patches seem
> to be working for you. They seem to solve the problem on my side; I
> can no longer reproduce the problem. Is it working for you?
I rebooted, but as I didn't know exactly how to trigger the bug except than
by waiting a few days I was waiting to see.
Do you have an idea about how to trigger the bug?
Regards,
Quentin Godfroy
On Wed, Oct 08, 2008 at 02:02:27AM +0200, Quentin Godfroy wrote:
>
> I rebooted, but as I didn't know exactly how to trigger the bug except than
> by waiting a few days I was waiting to see.
>
> Do you have an idea about how to trigger the bug?
The bug is that each time a transaction commits, if the buffer head
hasn't been leaked already, it will leak pinning the memory until the
total size of the journal is taking up memory. If you have a 2gigs of
memory, and a single filesystem with 256meg journal, you might not
notice the problem at all. If you less than 256 megs of memory,
you'll notice fairly quickly.
So you can test to see if you are running into problems by running the
buffer_dump program to trigger a dump of buffers in use. Some buffers
will always be pinned, such as the super block, block group
descriptors, and journal super blocks. And there will be some number
of buffers that are in use; but the number of dirty buffers should be
no more than, oh, 30 or 40, and over the next couple of hours, it
should not be growing. Even before you start runninng to memory
exhaustion problems, the number of in-use buffers should relatively
quick rise to hundreds or thousands after a few hours of run-time.
Regards,
- Ted
On Tue, Oct 07, 2008 at 08:53:38PM -0400, Theodore Tso wrote:
> On Wed, Oct 08, 2008 at 02:02:27AM +0200, Quentin Godfroy wrote:
> >
> > I rebooted, but as I didn't know exactly how to trigger the bug except than
> > by waiting a few days I was waiting to see.
> >
> > Do you have an idea about how to trigger the bug?
>
> The bug is that each time a transaction commits, if the buffer head
> hasn't been leaked already, it will leak pinning the memory until the
> total size of the journal is taking up memory. If you have a 2gigs of
> memory, and a single filesystem with 256meg journal, you might not
> notice the problem at all. If you less than 256 megs of memory,
> you'll notice fairly quickly.
Indeed after a couple of days of uptime the number of dirty blocks do not go
further than ~50, so I think the bug is corrected as far as I am concerned.
By the way, why does the kernel not commit to memory these remaining buffers
when the memory is scarse (say instead of firing an OOM killer)?
Regards
Quentin Godfroy
On Thu, Oct 09, 2008 at 01:52:46AM +0200, Quentin Godfroy wrote:
>
> Indeed after a couple of days of uptime the number of dirty blocks do not go
> further than ~50, so I think the bug is corrected as far as I am concerned.
>
> By the way, why does the kernel not commit to memory these remaining buffers
> when the memory is scarse (say instead of firing an OOM killer)?
The bug was the jbd2 code was bumping the reference count for the
buffers, so they were considered "in-use" and thus couldn't be freed
and released for other uses.
Regards,
- Ted