Received: by 2002:ac0:a581:0:0:0:0:0 with SMTP id m1-v6csp726238imm; Wed, 20 Jun 2018 05:46:12 -0700 (PDT) X-Google-Smtp-Source: ADUXVKIIePl62LSdbktNN3SkkRb1BlDFsXqPsZxeshiQOy8oX/TuQKi6hlPmH9mElsyJ6I9J672y X-Received: by 2002:a17:902:8308:: with SMTP id bd8-v6mr23775303plb.329.1529498772574; Wed, 20 Jun 2018 05:46:12 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1529498772; cv=none; d=google.com; s=arc-20160816; b=lZGJg2DRYX7MTvDHa0o2/Puc8G9vYXmb6xDAbF/JCvI8iQ8mXCPZjYLYyernCy2Hfv ocyrEZXPZq0fH1jDHDSdKmUv6gTjOhdJegVjRWzXEX+++Xd5WlR4omkQPNIi44sD4Meo 0XtGW89DngslHbxVaJ8+IjJSLo53ZjrXvH3XUB2Wq7vVBNuehDQ+bF5N3G0VKCCRj22a QBWR7pKBIlFKxe/DKXsx/aSJXCm82d22qu5YWF96Vmu6kbjJTejl+uahGUGHw7cFa1eV w7zxL/jojplVWW314axT1PcIFZhKYnjyBMVsrPJSf06HvJ7swizNFpFNvF5CSo27wc6i 5utg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding :content-language:in-reply-to:mime-version:user-agent:date :message-id:autocrypt:openpgp:from:references:to:subject :dkim-signature:dkim-signature:arc-authentication-results; bh=raELitttRMG27jwSLadN9z75RtXgIRVAO332BZaXAfU=; b=gI41SB/cJ4pkAhK0Ct014zmN258x2IRvMRqxMiiKxTiIU/zxjKYczB6THVS1sTMi0+ 0BbSk3MBudhmcR4ai9aQXKbDr6Mga4QK3g8oNBVJT+2Dk9uzKLlg2ZzbHzLjzil51DoS Ot+vcZPI3jYsIM0+sn7Lb60RXrXJes9DyTeUPSxt6yNs3oNDxi4TfSFRwhnobyuJbJfT Cny9oxa6SytMBD7w0RkjQkhTvZxkAUCpThCAAv2kVsTE9mlxzr12YVOqW2RkNkZD/uL2 k7fbmJV0dGf0aG///GzVeixpBB1t45HI5aZ9wOugyoFlEPR9MlYAlLbXERhe40aBGX5F MM6g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@boo.tc header.s=fm3 header.b="GcF9/lu6"; dkim=pass header.i=@messagingengine.com header.s=fm3 header.b=QNpovutC; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id a19-v6si1894499pgv.47.2018.06.20.05.45.57; Wed, 20 Jun 2018 05:46:12 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@boo.tc header.s=fm3 header.b="GcF9/lu6"; dkim=pass header.i=@messagingengine.com header.s=fm3 header.b=QNpovutC; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752884AbeFTMpN (ORCPT + 99 others); Wed, 20 Jun 2018 08:45:13 -0400 Received: from out2-smtp.messagingengine.com ([66.111.4.26]:52693 "EHLO out2-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750852AbeFTMpK (ORCPT ); Wed, 20 Jun 2018 08:45:10 -0400 Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 5388421E3C; Wed, 20 Jun 2018 08:45:09 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Wed, 20 Jun 2018 08:45:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=boo.tc; h= content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-sender :x-me-sender:x-sasl-enc; s=fm3; bh=raELitttRMG27jwSLadN9z75RtXgI RVAO332BZaXAfU=; b=GcF9/lu665nQ/M65I1F/Dh/MPLgRbT/pKrxJNoH2IYQ3G 60cbvCLV30a2yon/RskkdNh+H3I1s3cNwZAntJEAWSmzA76RthMH6/DA74tcRKG4 fAYtASo3zXS8GLw3hANejF3YZy/PosumryZtKV9L+UCjBVhGqPV2ctB7bqlQV93L hLf3TcigVym/L1J6SEUfAvBaqAmS9Nfg0Rth39ByKu5+ixjB+ChLWCutUUEPnrKL 8FWHNGXD7lA+IfEt8EfLIeO5OFMR5/NDHSikCGTVzLfFZaPxb1i/1NRTEDz0D8Mo twQQ/yj0UMzhgbSDgQEpS8LVg13ajs8cgDqKXcC/g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=raELit ttRMG27jwSLadN9z75RtXgIRVAO332BZaXAfU=; b=QNpovutCh+SUfxoOKo3W9X IORvXloMQM3bhC+66vZ7ExtJi+zqn0wGKmMKd+CtkYwwtkm7takemmmwfZy2cMkr e2msRUBuvFK7bbBYKBpJzkZ5IRbNxdWqzaF/1IB446vh3GJtKHP2KWN2pExX8Pl2 XCV14+FM/RRqJ2sJC3ZKu0FMAEenTHUqps60QsS62Qw2HiEDJGpVrYS+v7uFU+Le RmoDhZKliBxUafbmW2zZDL0MGtZSApRD1VuC0FQoe2TAvAabr2+4IHP67kFEghKU OJq1jhMNtNnHzYfOit3q0b3wq82hq8bcmZvN/rvJ6NYYFJFEFkDkUBVsWQyUJUaQ == X-ME-Proxy: X-ME-Sender: Received: from [10.90.90.106] (2.162.187.81.in-addr.arpa [81.187.162.2]) by mail.messagingengine.com (Postfix) with ESMTPA id 7EBFF1025D; Wed, 20 Jun 2018 08:45:08 -0400 (EDT) Subject: Re: Hard lockup in blk_mq_free_request() / wbt_done() / wake_up_all() To: Jens Axboe , linux-kernel@vger.kernel.org, linux-block@vger.kernel.org References: <9788e0e6-a448-bf85-1f41-88f42dc0071d@boo.tc> <7080a91c-8d9a-6305-2b67-dc27a374327a@boo.tc> From: Chris Boot Openpgp: id=846753CB19213142C56DC918F5C83C05D9CEEEEE Autocrypt: addr=bootc@boo.tc; prefer-encrypt=mutual; keydata= xsFNBFL1FNgBEADf8jZGW5tZWPDpyx7oWq8L7KD9a2YM5bp48LJ9tXYEVD+j3EIJH3DlYMOh Lif5+XkMaHNAakXSbo41Sjf3ArYOz+ZNvpR3ln/kqYv/ntgbAstlWuWLxGJbjJuLxjSh1eU5 jn+XAr0OvQMO9DiwBN3Ocm5B6tkUNhasxOmdlAxef0FsK7Y5bbqxVjC5/3DHqbmDiJvdof4q 1z5SEpuzKLn5xmdU+kANurZekp0JqgprS8gSmDV3fpJa7gTmcX11ArAV4TbI5CmJgnv3u6Nf k8E6oLk7wDs6mKzutS1MMVtaWpOMYqbM8q/QFI+ICf5SGmvpvOTvgIxAC80RWTYaxZn0g6sQ BhnByDcXFk/YYncmbHBYRJBbb+Y5lRGJMiv7KIp0BzDHO2zcDqvAiC2mtEl+iDOC06vqMD+t YRMkjtDsHbB7TCEeFmeSrQddLfoce04cnl3AyY22Vp2J2GsfobdX2Jw1drBou9cUN7shpuCU cqcGEvpT6mRd6uIzbFNXkWp0wiQPKUzDJXlh/GiROtM/468Bbj9JsiIIv183iKw6fQJtMg5c B34/GuEFfbfrqPNNO2ElEX6DcsnRZp3Vq+SMM+dDWXYSF1MJt52tT+deHGgzXj+NMHWU/K5X DWGcxtpM8QbFFwxTl2B5k2jjL61IhCnPpJSQZhzhXRuei04uaQARAQABzRlDaHJpcyBCb290 IDxib290Y0Bib28udGM+wsGXBBMBCgBBAhsDBQkJNRFnBQsJCAcDBRUKCQgLBRYDAgEAAh4B AheAFiEEhGdTyxkhMULFbckY9cg8BdnO7u4FAlnvVCECGQEACgkQ9cg8BdnO7u7HjhAAkug/ zY8fezVbbJr3qaNp/PtLi6a5V2zWCY9rsKach3C5oPsbdQs3Y6Lu8y79RcTjVkuwA+K6kB7J Jw/ScoN3+a+cPCnegE5G+fup7wXeQFBO62VGFDjS3+fyLJ/EV0ou04BIo65i1vwev3F6WUBJ X3FULUsIACI1b29RqIGQPcnxmZar5X0yCsoSKUEPHvrNgqhty9nVbugIIhsn0CRfN1oOCWwj skW2GH7fAF3enaNfoEJgijwVK5WBHNpj9AbDZIscpY2GaQKHD9sYC3efJMNllUXeA92CNilt T+sZWnEmz9I2Kt6M85N2MRuZ/Zoa9bPphVyhsVoTdG28yBk+XkUm2aCR2R27WmmzDH06LMt/ k52m+v5sYygy9LURyWyLGcE7HMXGQCjnbWziezSwQN8uj/cqqmIk/Z2PJmHPqeQOUsZIQelM 7ykM/JX1AUo6Bk7y/JvzD49Ry3Qn8disN4bL1WhuaxKWH7kHC6ivHZag7oPKGGI/SPK9NddD CchQzrwrNS2C0Ik5Gp81WKNFaE2WDa2BanMaaKP7reSvD2abjThZ51YyyEhPSOIEf0HU5/dy VIMOgMtaFcRx7InV4fSfqQTM/JRzbUTb2DfeVV5HMIK5+VuCsLq0GRG03fWvfyuA841SvwaI 7MXwoAX1dbSzkdIvNveUQ6k1pPb99xvOwU0EUvf4ogEQAKkdFtOZUfNQIWGAuJfYOTnoLqqC kre6E0kw18DpXlH97O+6lKPLB679pKMfzh7uwVlkIjWwc0gQPxQvmKv6PbkflAMzr7FtofNj fMi1eaGdSlRAbo2K1EQTukVTtnkPFOd+Xgp74Gq+Ebr73qO3on04wvM6NzzBdLh+QEWxj4WC Jv6/Eh3BWiyOTAS3qyL1pZiqorrXhmBu4WvoaR2+AgasOVV1d0+flmbj7OQIieQtORLadyyH 7a/c/Q+h+9Dabt6BNT2IdOMEkMm61tdOCsqg2MgsgTyU8FjSnJE+cws/H1W1aufCldD47dpN bJHawl7WEVYYoABuApvXTi6DLNWql0v0ownhNwVKZb3zs/AdkoDRjYb9YSQ/WIPcNtiGrr3p 6xeIKr93EuqZWtWvtpF5DqoJ7FNqN5wQEmOlpj7igQ0r9M3tTQQJg0j6MtCdbo9ZUXtZmjxi 8mdpAz0of8qabgSiPhFuFgHDnqGtRmVgKCY1vD6esmA+wfZnbGaU0tmQQpr2Cdbx11vnfhj/ LTObPBYy+ciJlPoXebC1/AsxANbLpjAtQUNWtXAS1NRFSuI1GtQ7RskqPS11uoRMhLkDy0aE 51QIQs3UWuTy591UGH8MwlNIy6pTjFCyRXeM2dynPzCECqOnZfyeuQ/dsiWInmDNRD1auGGE F+Faf11dABEBAAHCwWUEGAEKAA8FAlL3+KICGwwFCQHhM4AACgkQ9cg8BdnO7u5pPw/9Hwp2 lFqViQ31CDeoebPNoS89EqQQV8IUm7qGd6f0OJx7pTa5iye/b7kNFDNGk/I+Secm1piUVybt iDvYe0vcRgTTQ14Oss41LZ6HK1oK8vxKDUSzPKTCQuONXkBNUI8o2WVQq9bsG5rU0Yj8ajfm J/a6cdJCgLdEr9zUaLXXIsitddwgchnqy1l/pNgpuDIQVHW10L30QWkOOLKpnNPfvw0qqctN a4/4I0rAnziTq7IzJntfr2/gBWEqfM5N5xT/luSh6YNar0lE8BupnqvtKNwZ+FwIA7i0M3mF QvgStHPQem5OVoFjpeimrFc19n2N7yxKavF1CtgxPOq0DYjj4g6CXwlWAjPqE7O/s0y6Dl1s S1jAyEaDEbjTskCscllClsdSabA/v3HOoMXerxRu6vo+icyvsZ/paGC/I8Rfk7QNv34Wme/T QYQs3zVTlnvc22XigtITDEzOq6ZUrPJ5SdWoL8pcpFsFI5hKHHJ0dmsNrrTXxV+7k2hkWxUk 5hb4iLTp0coX36cYtERw73Z5kQHl6TEcCbhhDjsNbTB3loMgBFrPRycOQloP0LhFbtm+ijYB FhHVl9ZfnKhf5G2HF/mR6NRZ0/KhhG8/VE71TIrJrdy6i3O4OXApqmVH58rkbvwQWNO+wQxL dGzE5kHmSn+1o9NBRfK3LEOZuVRUd4fCwWUEGAEKAA8CGwwFAlTbJUYFCQPEYCQACgkQ9cg8 BdnO7u5kkw//fuQjpl0P3JGtxvO+qwxnYN2+/+Ciwe5muansVowGLC/vnh5if40joqykd/pH T5igm+nPsfTYUrhsvusmf4exhJOeOQgLsR4Q3xjFfi1EsO9J6GZFMMajsoVLYci4AVkfCZFE o7iSzlFDu4Dsr2YiafaJa3DiNn864T9HRAjnkCg2YjvCfl+i2En9tLTnsBoZSxVVmBuA4kt0 dXOUJAgWWix4OY+ScdO9UobGi/Nb7T9cN/MNXJnmhqfa7kfsK6JWGRnyL8FsuWemNJt9BbdN WNIS9amrEzR4hm0o6BvjQPyDixiPvSxXiLVnI6xM+lrYst+l5gLwq3TfdhkgjpzwTJdAWgnm BQD1hCrN/kkhckzD8ga0meGUt4ulcDG4iksM+/DehLqmzfXdUSKG+8gtCZEcf8g3bogaXXjH JXUZdIfU1AUld+nOy52zccEkmuoi1HGKdbhvOhESqMq797DzE8Oyex9ghBLUibyqmsh2rEwO 1zeSijuTu7uWs4FyRH6pJJYU/RYc+YjKTh6vUsYsa7H69PxmW0Odskf4tIV4NPn2gU5ULUJ2 is9OTJbq6qXtc1+DiqPja1dGMuJ6QNpbOTOaF9+MxHRdCmDcKLOAzQWsdlJl4b7y7KoLx22v 1xqpV6LNZL4Ou2RnkZmUHEbHhH5kQ0/MkHS+tWyXQWKeJpLCwWUEGAEKAA8CGwwFAlZ9sWQF CQVwJsIACgkQ9cg8BdnO7u5lExAAzSk9NiX/OSqonv8T4H9XC8b3MHWhKadJU33I71J4n1or 5XZQRSruzVJO2tAHdFOHo0hH7fbKnwVQxbcWEdMd+khtqMOIR0yRMpiKNFs0CrkNCNHhiYyO axTjgNGxg7KudQcJl200WkFMQKLmE+EzW1Wt3fk2Hkd63Oi2WfXQTvVm9AHtgrBSs6ZJ6+rz hNDc+yw5V+/MAzkrRk2UPq7pbTu1cHek5p3sK3p9Y15TyF7+A+nT4vxE5rL6Ki1GMxRjJ3tx zWozdDEjRrPmTdZJ6mPr4oHAjmn140vh813mMxOD4M4eQUNlQb+97cb1dhjst6Jft96Cs/gp Tne/KKL7vIioeCSjLzJ7t1v6hIP8IxemWdbba3b2/yB2x8luJ2ezYauWUNQ0JBy/iXXYm+TN nMcuY79n6hyf7SvZxaYyV+6M5tU2ouoFXVDNFH0xIp9AWqSAGBHqXIaEoXDFve5XiX5g0+tq bof5bJEJ1fk1lnMu27Y2EhQRp2hvRHJ3Gxe1mPDKQfetbXJ9IvNmMrASlrgslbUhtH0xXpRX xfBy0t5exdLUSYkKsnI0PYiJMdnFY4rbHAVMLjFPuGWIxC7/4/yMnbHIGfPFdCXPtfFMahBT hTiQauIBAZ7uCLicUDri1x0ZCx7NMAyiMTZVadiAWdrBxjZq0mmr4VklTGGzl0jCwXwEGAEK ACYCGwwWIQSEZ1PLGSExQsVtyRj1yDwF2c7u7gUCWEaAsQUJB1CxDwAKCRD1yDwF2c7u7sWI EADb2XOxW2zsOUP0YHG2g6IWT6OF8vQJTxm16hCoDnhSJkjq77jY8u0kkJ6NFAZzUAfTxNRT LL8RMylnC+q662habHmQWi3jhUYx8HsKVCpibd4Z9//A3yikiaK4I8tPg7Izgvo9CPS2pr65 fH9DSBJxc9kLVw7lbhZeIwHxsb+WS/B8qks3c9UjtqZo+mzF+JO+J/8nS7dLr/R0l+c7Egu6 LYLhkDGeIiA/8nH6fjDDPZ6fCiZvnBOeUYxts2arsGqMgCtxkpx6x+t+eK2xokLRnrUqoq6r zEBa+7hFpqfvjXSUQgMheZ86j+YuUW0+v8Y8GbAUyiSTKHZtIChpGL1C00AzWTsE+9VPRG7E t4EhyI7+RU22XbN8AjFlAjYtpORCbJhFSUCs6AjEc3Nr+49dEuwlb23ZaleMqubG+WNnZJxk 2cBUsL97uQuDl2tfh4PAIthNYpbDKrd2iozM9Nu0hiDZ+934aYZH3sTvXXSLta92ZLmO4dJY fBxvmaDVItKllPeOmdST+AhYrdgMoM1TsmaHjQ3RQRqkWgPDcQMj0BQy/LxbnWirFuah2hVy bhEr0Eq0s9WJXEBWtTA+4ODtyO6iXr8oeKuWz5iJXMGgjWXGITDzAqA6D/R/1cQp4aC1ngQP ku3XuP/WF5462wPDOBuwNTEotsffSI99uW7np8LBfAQYAQoAJgIbDBYhBIRnU8sZITFCxW3J GPXIPAXZzu7uBQJZkbIyBQkJMi2LAAoJEPXIPAXZzu7uyH8P/2ZQ3HLncqpOBEn9faQONdsz cJri3utPfJbOReUTI4AE+l8LSx9/gwg/+0WUXuF/e4XM66L6fGDevIgHWcSfdyY9rHQz+saZ PJS36mU0gM0ZXnhJ+ooTmBgd3vQ5OLGUO3azbbtGUvzYMhpGFDtq+mKKQG4op3HwMDA4RW6o qp0vE8oB1BDzUXYra6k6EPuihP/qbVm94Z5t1s3EuNQAZaH8dtjn7R561v9RSZ0sJ3p4V1sT OQmbwLgUc4ZOoXFsAXg1rDikzQqMOKPNEszGrNYikSmcrwlAq7Qs95MRxsc1cu6F6xFYVHiF wX+Z47OCmlWjl+uDTuVhuGBUAUbcVX+yZVapCy/b/gYmg3/9GYvHnd4FRuM/xeGnU6Ejwugg oS0bF/BlNbeSEr4zqjQM7LqcbX70XklJANKNLJfSuTSS2I351ix45dV5J2jBgqV91q+t1pmp KCHsI9SMw4g4FzAqduLZzHa9UiTxPMe/bGb6Y/ma3ErgCznAGXGhhXnPCOCDO78t4rf1CXda EVg5Zyqa7zzzbB2rNWcTnjJQQUHMa/pindxmYZdsdvE2YBMW0JyaNvI/u829WIXHS1oRU1so f71N3bU6BxRWU54d+cpEae0hYO2DxcxVTkq+lcQpoIzZLxzpsnUuIFr+EtXm3Y4apdjGHg54 O6CmtAEX0Sww Message-ID: <9c444ab8-2e50-c42a-dae1-86954358218e@boo.tc> Date: Wed, 20 Jun 2018 13:45:07 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-GB Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 12/06/18 17:22, Jens Axboe wrote: > On 6/12/18 10:19 AM, Chris Boot wrote: >> On 12/06/18 17:09, Jens Axboe wrote: >>> On 6/12/18 9:38 AM, Chris Boot wrote: >>>> Hi folks, >>>> >>>> I maintain a large (to me) system with 112 threads (4x Intel E7-4830 v4) >>>> which has a MegaRAID SAS 9361-24i controller. This system is currently >>>> running Debian's 4.16.12 kernel (from stretch-backports) with blk_mq >>>> enabled. >>>> >>>> I've run into a lockup which appears to involve blq_mq and writeback >>>> throttling. It's hard to tell if I've run into this same thing with >>>> older kernels; I'm trying to track down a deadlock but so far I've been >>>> fairly certain that involved the OOM killer, but this doesn't seem to. >> [snip] >>> >>> Hmm that's really weird, I don't see how we could be spinning on the >>> waitqueue lock like that. I haven't seen any wbt bug reports like this >>> before. >>> >>> Are things generally stable if you just turn off wbt? You can do that >>> for sda, for instance, by doing: >>> >>> # echo 0 > /sys/block/sda/queue/wbt_lat_usec >>> >>> It'd be interesting to get this data point. Eg leave blk-mq enabled, and >>> then just disable wbt. >> >> Hi Jens, >> >> Thanks for the speedy response. I'll see if I can get that tested soon; >> if the system is stable without blk_mq I can see the users wanting to >> keep it that way for a while. I'll let you know. > > Understandable. I just get suspicious of the general state of the system, > if it's locking up there. Could be a hardware issue, or a bug in some > other area that's messing things up. I have wbt running on literally > hundreds of thousands of boxes and haven't seen a lockup like this. Hi Jens, I got an opportunity yesterday to do some testing. I can't get this system to crash with blk-mq disabled, or with blk-mq enabled but wbt disabled. I have a reproducer workload I can launch against the system and it seems to crash reliably with this, but I doubt I can share it with you. I do, however, have a task state dump (SysRq+T) that I managed to get out of the server once it started locking up. It's pretty large, so I uploaded it to my Dropbox for now: https://www.dropbox.com/s/fyo1ab6mmcqk8fq/crash-1.log.gz?dl=0 Hope this helps! Cheers, Chris -- Chris Boot bootc@boo.tc