2014-10-31 12:28:36

by Jeff Layton

[permalink] [raw]
Subject: [PATCH] lockd: ratelimit "lockd: cannot monitor" messages

When lockd can't talk to a remote statd, it'll spew a warning message
to the ring buffer. If the application is really hammering on locks
however, it's possible for that message to spam the logs. Ratelimit it
to minimize the potential for harm.

Reported-by: Ian Collier <[email protected]>
Signed-off-by: Jeff Layton <[email protected]>
---
fs/lockd/mon.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/fs/lockd/mon.c b/fs/lockd/mon.c
index 9106f42c472c..1cc6ec51e6b1 100644
--- a/fs/lockd/mon.c
+++ b/fs/lockd/mon.c
@@ -214,7 +214,7 @@ int nsm_monitor(const struct nlm_host *host)
if (unlikely(res.status != 0))
status = -EIO;
if (unlikely(status < 0)) {
- printk(KERN_NOTICE "lockd: cannot monitor %s\n", nsm->sm_name);
+ pr_notice_ratelimited("lockd: cannot monitor %s\n", nsm->sm_name);
return status;
}

--
1.9.3



2014-10-31 12:36:18

by Trond Myklebust

[permalink] [raw]
Subject: Re: [PATCH] lockd: ratelimit "lockd: cannot monitor" messages

On Fri, Oct 31, 2014 at 2:28 PM, Jeff Layton <[email protected]> wrote:
> When lockd can't talk to a remote statd, it'll spew a warning message
> to the ring buffer. If the application is really hammering on locks
> however, it's possible for that message to spam the logs. Ratelimit it
> to minimize the potential for harm.
>
> Reported-by: Ian Collier <[email protected]>
> Signed-off-by: Jeff Layton <[email protected]>
> ---
> fs/lockd/mon.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/fs/lockd/mon.c b/fs/lockd/mon.c
> index 9106f42c472c..1cc6ec51e6b1 100644
> --- a/fs/lockd/mon.c
> +++ b/fs/lockd/mon.c
> @@ -214,7 +214,7 @@ int nsm_monitor(const struct nlm_host *host)
> if (unlikely(res.status != 0))
> status = -EIO;
> if (unlikely(status < 0)) {
> - printk(KERN_NOTICE "lockd: cannot monitor %s\n", nsm->sm_name);
> + pr_notice_ratelimited("lockd: cannot monitor %s\n", nsm->sm_name);
> return status;
> }
>

How is this being triggered repeatedly? Normally, the 'cannot monitor'
message should be happening at client mount time or not at all.

--
Trond Myklebust

Linux NFS client maintainer, PrimaryData

[email protected]

2014-10-31 12:44:03

by Jeff Layton

[permalink] [raw]
Subject: Re: [PATCH] lockd: ratelimit "lockd: cannot monitor" messages

On Fri, 31 Oct 2014 14:36:16 +0200
Trond Myklebust <[email protected]> wrote:

> On Fri, Oct 31, 2014 at 2:28 PM, Jeff Layton <[email protected]> wrote:
> > When lockd can't talk to a remote statd, it'll spew a warning message
> > to the ring buffer. If the application is really hammering on locks
> > however, it's possible for that message to spam the logs. Ratelimit it
> > to minimize the potential for harm.
> >
> > Reported-by: Ian Collier <[email protected]>
> > Signed-off-by: Jeff Layton <[email protected]>
> > ---
> > fs/lockd/mon.c | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/fs/lockd/mon.c b/fs/lockd/mon.c
> > index 9106f42c472c..1cc6ec51e6b1 100644
> > --- a/fs/lockd/mon.c
> > +++ b/fs/lockd/mon.c
> > @@ -214,7 +214,7 @@ int nsm_monitor(const struct nlm_host *host)
> > if (unlikely(res.status != 0))
> > status = -EIO;
> > if (unlikely(status < 0)) {
> > - printk(KERN_NOTICE "lockd: cannot monitor %s\n", nsm->sm_name);
> > + pr_notice_ratelimited("lockd: cannot monitor %s\n", nsm->sm_name);
> > return status;
> > }
> >
>
> How is this being triggered repeatedly? Normally, the 'cannot monitor'
> message should be happening at client mount time or not at all.
>

No, it's triggered on NLM lock activity. I don't think we really do
much with NLM at mount time, do we? In any case, the bug was reported
here:

https://bugzilla.redhat.com/show_bug.cgi?id=1155232

In this case, it was an application that was apparently responding to
ENOLCK errors by retrying the lock.

--
Jeff Layton <[email protected]>

2014-11-06 19:48:34

by J. Bruce Fields

[permalink] [raw]
Subject: Re: [PATCH] lockd: ratelimit "lockd: cannot monitor" messages

On Fri, Oct 31, 2014 at 08:28:29AM -0400, Jeff Layton wrote:
> When lockd can't talk to a remote statd, it'll spew a warning message
> to the ring buffer. If the application is really hammering on locks
> however, it's possible for that message to spam the logs. Ratelimit it
> to minimize the potential for harm.

Thanks, applying for 3.19.--b.

>
> Reported-by: Ian Collier <[email protected]>
> Signed-off-by: Jeff Layton <[email protected]>
> ---
> fs/lockd/mon.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/fs/lockd/mon.c b/fs/lockd/mon.c
> index 9106f42c472c..1cc6ec51e6b1 100644
> --- a/fs/lockd/mon.c
> +++ b/fs/lockd/mon.c
> @@ -214,7 +214,7 @@ int nsm_monitor(const struct nlm_host *host)
> if (unlikely(res.status != 0))
> status = -EIO;
> if (unlikely(status < 0)) {
> - printk(KERN_NOTICE "lockd: cannot monitor %s\n", nsm->sm_name);
> + pr_notice_ratelimited("lockd: cannot monitor %s\n", nsm->sm_name);
> return status;
> }
>
> --
> 1.9.3
>