Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
MIME-Version: 1.0
References: <20181216190639.6safwjqwdphkce67@gmail.com> <20181216194649.GA7732@pure.paranoia.local>
 <20181216195343.idnt2y5y5wjky5gu@gmail.com> <20190104013522.stng6gwauwnr6wbi@starla>
 <CAErSpo5a2oO=5byEuA5AouS=kBmj7ihw2EVYAvJcdti29Tf1HQ@mail.gmail.com> <20190305232600.GA12110@dcvr>
In-Reply-To: <20190305232600.GA12110@dcvr>
From:   Bjorn Helgaas <bhelgaas@google.com>
Date:   Wed, 6 Mar 2019 14:50:20 -0600
Message-ID: <CAErSpo6QTdABwBx-nvkmJyv7JHb95azapwLtBf++Ved7=UpMWA@mail.gmail.com>
Subject: Re: [RFC] LKML Archive in Maildir Format
To:     Eric Wong <e@80x24.org>
Cc:     Joey Pabalinas <joeypabalinas@gmail.com>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        kernelnewbies@kernelnewbies.org,
        Linus Torvalds <torvalds@linux-foundation.org>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        Konstantin Ryabitsev <konstantin@linuxfoundation.org>,
        Eric Biederman <ebiederm@xmission.com>,
        Jasper Spaans <j@jasper.es>
Content-Type: text/plain; charset="UTF-8"
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Tue, Mar 5, 2019 at 5:26 PM Eric Wong <e@80x24.org> wrote:
> Bjorn Helgaas <bhelgaas@google.com> wrote:

> > Any pointers?  I guess there's no mutt backend that can read a
> > public-inbox archive directly?
>
> There's mutt patches to support reading over NNTP, so that
> works:
>
>         mutt -f news://$INBOX_HOST/$INBOX_NEWSGROUP

Neomutt includes NNTP support, so I tried this:

  neomutt -f news://nntp.lore.kernel.org/org.kernel.vger.linux-kernel

which worked OK but (1) I only see the most recent 1000 messages and
(2) obviously isn't reading a *local* archive.  Neomutt took about 45
seconds to start up over my wimpy ISP.

I assume I could probably have a local archive and run a local NNTP
server and point neomutt at that local server.  But I don't know how
full-archive searching would work there.

> I don't think mutt handles mboxrd 100% correctly, but it's close
> enough that you can can download the gzipped mboxrd of a search
> query and open it via "mutt -f /path/to/downloaded/mbox.gz"
>
>   curl -XPOST -OJ "$INBOX_URL/?q=$SEARCH_QUERY&x=m"

I got nothing at all with -XPOST, but this:

  curl -OJ "https://lore.kernel.org/linux-pci/?q=d:20190301..&x=m"

got me the HTML source.  Nothing that looks like mboxrd.  I assume
this is stupid user error on my part, but even with that resolved, it
wouldn't have the nice git fetch properties of the git archive, i.e.,
incremental updates of only new stuff, would it?

I think my ideal solution would be a mutt that could read the git
archive directly, plus a notmuch index.  But AFAIK, mutt can't do
that, and notmuch only works with one message per file, not with the
git archive.

Something that might work would be to use Konstantin's "git archive to
maildir" hint but shard into a bunch of smaller maildirs instead of
one big one, then have notmuch index those, and use mutt or vim with
notmuch queries instead of having it read in a maildir.

But I feel like I must be missing the solution that's obvious to
everybody but me.

Bjorn