path: root/Documentation/piem.texi

\input texinfo @c -*-texinfo-*-

@set VERSION 0.5.0

@setfilename piem.info
@documentencoding UTF-8
@documentlanguage en
@settitle Emacs tools and glue for working with public-inbox archives

@copying
Copyright all piem contributors <piem@@inbox.kyleam.com>

Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3 or
any later version published by the Free Software Foundation; with no
Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.  A
copy of the license is included in the section entitled ``GNU Free
Documentation License''.
@end copying

@dircategory Emacs
@direntry
* piem: (piem).  Emacs tools and glue for working with public-inbox archives
@end direntry

@finalout
@titlepage
@title piem reference manual
@subtitle for version @value{VERSION}
@author Kyle Meyer
@page
@vskip 0pt plus 1filll
@insertcopying
@end titlepage

@contents
@*

@ifnottex
@node Top
@top piem

This manual is for piem version @value{VERSION}.
@end ifnottex


@menu
* Overview::
* Getting started::
* Applying patches::
* Miscellaneous functionality::
* Contributing::
* Related projects and tools::

Appendices
* GNU Free Documentation License::  The license for this documentation.

Indices
* Key Index::
* Variable Index::
* Lisp Function Index::
* Concept Index::
@end menu


@node Overview
@chapter Overview

piem is a collection of Emacs libraries for working with public-inbox
archives.  As much of the hard work here is already done by other Emacs
libraries---things like mail clients, news readers, Git interfaces, and
even web browsers---piem is mostly about bridging some of these parts
for convenience.

@node public-inbox
@section public-inbox
@cindex public-inbox
@cindex lore

@url{https://public-inbox.org/README.html,public-inbox} is software for
archiving public mailing lists.  Archives can be exposed over HTTP.  As
examples, @url{https://public-inbox.org/meta} serves public-inbox's own
mailing list, and @url{https://lore.kernel.org/lists.html} hosts the
archives of many Linux development mailing lists.

@cindex pull methods
These web archives are good for searching, particularly if you don't
have all of the list's messages on your local machine, or for linking to
a message.  On the other hand, the web interface isn't convenient when
you want to follow new activity on a list.  To do that, you could of
course subscribe to the mailing list, but public-inbox offers a few
``pull methods'' that you can use instead:

@itemize
@item
an atom feed for the list as a whole or for specific searches
@item
read-only NNTP
@item
read-only IMAP (new in upcoming public-inbox v1.6.0)
@end itemize

Finally, archives are exposed as one or more Git repositories,
facilitating replication (see
@url{https://public-inbox.org/reproducibility.html}).  For example, you
can clone the mailing list archives of @samp{git.vger.kernel.org} with

@example
git clone --mirror https://lore.kernel.org/git/0 git/git/0.git
@end example

@noindent
After the initial clone, new messages can be retrieved with
@code{git fetch}.  Unsurprisingly @code{git log} is not a pleasant way
to read a mailing list; instead this method is useful for mirroring the
archive or bulk importing of the messages.  (See
@url{https://public-inbox.org/clients.html} for a list of some tools
designed to work with public-inbox archives.)


@node Getting started
@chapter Getting started
@findex piem-dispatch

@code{piem-dispatch} transient
(see
@ifinfo
@ref{Top,,,transient})
@end ifinfo
@ifnotinfo
@url{https://magit.vc/manual/transient/})
@end ifnotinfo
provides an entry point to piem commands.
It's recommended to bind @code{piem-dispatch} to a key.  However, before
most of those commands do anything useful, you need to register inboxes
and activate at least one minor mode.

@node Registering inboxes
@section Registering inboxes
@cindex coderepo
@cindex inbox
@vindex piem-inboxes

A public-inbox archive, referred to as an @dfn{inbox}, is registered by
adding an entry to @code{piem-inboxes}.  Here's an example entry for the
Git project's mailing list:

@lisp
("git"
 :url "https://lore.kernel.org/git/"
 :address "git@@vger.kernel.org"
 :listid "git.vger.kernel.org"
 :coderepo "~/src/git/")
@end lisp

@noindent
The first element is a name for the inbox and will typically match the
name at the end of the @code{:url} value.  Specifying either
@code{:listid} or @code{:address} is important so that a message in a
buffer can be mapped to an inbox in @code{piem-inboxes}.

@code{:coderepo} points to the working tree of a local Git repository
that contains code related to that archive (in the example above, a
local clone of @url{https://git.kernel.org/pub/scm/git/git.git/}).  This
information is required to apply patches from an archive to a local code
repository (@pxref{Applying patches}).  To map an inbox to more than one
repository (e.g., if the inbox receives patches for multiple projects,
or if you use a few dedicated Git worktrees for different types of
patches), set @code{:coderepo} to a list of locations.

@cindex issue tracking
@cindex Debbugs, issue tracking system
When using the Debbugs instance at @uref{https://bugs.gnu.org} to
retrieve messages, an inbox can mapped to each message by setting
@code{:gnu-package} for an inbox.

@findex piem-merged-inboxes
@vindex piem-get-inboxes-from-config
If you mirror some inboxes locally (e.g., for fast local access or for
use with lei), you don't need to duplicate the information from your
public-inbox configuration
(@url{https://public-inbox.org/public-inbox-config.txt,public-inbox-config(5)}).
When the option @code{piem-get-inboxes-from-config} is non-nil, the
function @code{piem-merged-inboxes}, which all code should use for
accessing the registered inboxes, returns a combined set of inboxes
derived from @code{piem-inboxes} and public-inbox's configuration.
Merging is done at the level of inbox properties (e.g., an inbox's URL
may be defined in @code{piem-inboxes} and the inbox's address in
public-inbox's configuration).  When a value is defined in both sources,
the one in @code{piem-inboxes} takes precedence.

Properties described for @code{piem-inboxes} are constructed by mapping

@example
[publicinbox $inbox]
$name = $value
@end example

@noindent
to

@example
  ($inbox :$name $value ...)
@end example

@noindent
The one exception is @code{:coderepo}.  In public-inbox's configuration,
the value of @code{publicinbox.$inbox.coderepo} points to another
configuration option, @code{coderepo.$value.dir}, which in turn points
to a repository's git directory.  The @code{:coderepo} of
@code{piem-inboxes}, however, should be set to the @emph{working tree},
so @code{:coderepo} is derived from the value of
@code{coderepo.$value.dir}, stripping a trailing @file{/.git} if
present.

@findex piem-clear-merged-inboxes
Note that @code{piem-merged-inboxes} reads from the public-inbox
configuration once, generates the merged set of inboxes, and then caches
the result.  If you change @code{piem-inboxes} outside the customize
interface or change public-inbox's configuration, you need to call the
command @code{piem-clear-merged-inboxes} to clear the cache.

@node Enabling integration libraries
@section Enabling integration libraries
@findex piem-debbugs-mode
@findex piem-elfeed-mode
@findex piem-eww-mode
@findex piem-gnus-mode
@findex piem-notmuch-mode
@findex piem-rmail-mode

With inboxes defined, the next step is to enable minor modes that teach
particular Emacs modes to link a buffer with a registered inbox.  piem
currently has libraries to support

@itemize
@item Debbugs
@item EWW
@item Elfeed
@item Gnus
@item Notmuch
@item Rmail
@end itemize

For example, if you use notmuch.el to read your mail, you can add
support for applying patches from a Notmuch message buffer by enabling
@code{piem-notmuch-mode} (@pxref{Applying patches}):

@lisp
(piem-notmuch-mode 1)
@end lisp

Help adding support for other modes, especially other mail clients, is
welcome.


@node Applying patches
@chapter Applying patches
@cindex am-ready mbox
@cindex applying patches
@cindex git-am

With @code{piem-inboxes} configured and appropriate integration
libraries enabled, a buffer that can be linked to an inbox can be mapped
to a code repository.  When reading a message in a
@code{notmuch-show-mode} buffer, for example, the list ID can be used to
identify the inbox and thus the associated local code repository.

There are two commands for applying patches:

@table @code

@item piem-am
@findex piem-am
This command tries to extract a patch from the current Notmuch or Gnus
message buffer and can handle an inline patch as well as one or more
patch attachments.

@item piem-b4-am
@findex piem-b4-am
This command relies on the b4 command-line tool to do more sophisticated
processing of the @emph{full thread} (e.g., pulling out the latest
reroll of a series) to generate an mbox that can be fed to @code{git
am}.  It is only compatible with inline patches.

@end table

@node Applying patches contained in a message
@section Applying patches contained in a message

@table @kbd
@findex piem-am
@findex piem-edit
@findex piem-edit-patch-am
@item M-x piem-am @key{RET} @var{branch} @key{RET} @var{base}
Apply the patch or patches in the current buffer to the associated code
repository.  Before applying, check out a new branch @var{branch}
starting at @var{base}.

Calling @code{piem-edit} instead of @code{piem-am} will pop up an
am-ready mbox for editing. To feed the edited mbox to @code{piem-am},
invoke @code{piem-edit-patch-am} (@kbd{C-c C-c}).
@end table

@findex piem-name-branch-who-what-v
@vindex piem-default-branch-function
You'll be queried for the name of the new branch.  The default name
offered is generated by @code{piem-name-branch-who-what-v}, which uses
the @samp{From:} and @samp{Subject:} headers to construct branch names
like @samp{km/b4-short-subj__v3}.  To use a different function to
generate the completion default, configure
@code{piem-default-branch-function}.

Next you'll be queried for the base to use as the starting point for the
branch.  If the sender specified a base commit for the series, that will
be provided as the default completion candidate.  Entering an empty base
signals to use the current branch of the repository as the base.

@vindex piem-am-create-worktree
@vindex piem-am-read-worktree-function
Rather than applying the patches directly to the associated code
repository, you can create a dedicated worktree by setting
@code{piem-am-create-worktree} to a non-nil value.  Giving a prefix
argument to @code{piem-am} inverts the meaning of
@code{piem-am-create-worktree}; that is, by default a prefix argument is
useful if you generally prefer to work within the configured code
repository but would like to trigger the one-off creation of a worktree
for a particular call.

@cindex magit
@vindex piem-use-magit
If the @code{piem-use-magit} option is non-nil (the default) and Magit
is loaded, piem uses Magit for some operations, particularly those that
are user-facing.  This includes jumping to the Magit status buffer for a
code repository after applying a patch.

@findex piem-am-ready-mbox
Note that the @code{piem-am} command works only for buffers from which
@code{piem-am-ready-mbox} can generate an am-ready mbox, which depends
on the enabled integration libraries.  Currently @code{piem-notmuch} and
@code{piem-gnus} implement the necessary functionality.

@node Using b4 to apply patches
@section Using b4 to apply patches
@cindex b4
@cindex lore

@url{https://git.kernel.org/pub/scm/utils/b4/b4.git,b4} is a
command-line tool for interacting with public-inbox archives.  While
useful for public-inbox archives in general, it is written for Linux
kernel development and focuses on the public-inbox archives hosted at
@url{https://lore.kernel.org}.

It's a fast moving target at the moment, but some of its current
capabilities include

@itemize
@item
downloading the mbox for a thread based on a given message ID
@item
extracting patches from a thread's mbox that can be fed to @code{git am}
@item
submitting and verifying cryptographic attestation for patches
@item
fetching a pull request found in a message ID
@item
generating a thanks email for patches
@end itemize

@noindent
The second item is the focus for piem, though at least some degree of
support for all of the above features will likely be added.
The entry point to applying patches with b4 is the @code{piem-b4-am}
transient.  (See
@ifinfo
@ref{Top,,,transient}
@end ifinfo
@ifnotinfo
@url{https://magit.vc/manual/transient/}
@end ifnotinfo
for more information on using Transient.)

@findex piem-b4-am
@code{piem-b4-am} offers the following actions:

@table @kbd

@item a
@itemx M-x piem-b4-am-from-mid
@findex piem-b4-am-from-mid
@findex piem-mid
@vindex piem-am-create-worktree
@vindex piem-am-read-worktree-function
Generate or download a thread's mbox for the current buffer's message
ID, process it into an am-ready mbox with b4, and then feed it to
@code{git am} called within an associated Git repository.  If a message
ID of the current buffer is not known (i.e. @code{piem-mid} returns
nil), one is read from the caller.  The caller is also queried for the
branch name and base, as described for @code{piem-am} (@pxref{Applying
patches contained in a message}).  And, as with @code{piem-am}, a
worktree can be created by configuring @code{piem-am-create-worktree} to
a non-nil value or by giving a prefix argument.

@findex piem-mid-to-thread-functions
To generate the input thread, first any functions in
@code{piem-mid-to-thread-functions} are tried.  This allows for a thread
to be retrieved from a local store (e.g., the Notmuch database).  If
that fails, try to download the thread from the @code{piem-inboxes} URL
associated with the current buffer, provided the current buffer's
message ID matches the one @code{piem-b4-am-from-mid} was called with.
As a last resort, call @code{b4 am} without a local mbox to let it
download the thread according to its own configuration.

@item i
@itemx M-x piem-b4-am-ready-from-mid
@findex piem-b4-am-ready-from-mid
Call @code{b4 am} with a given message ID.  This differs from
@code{piem-b4-am-from-mid} in that it is a direct wrapper around a
command-line call to @code{b4 am}.  The caller is always queried for the
message ID, and the final product is an am-ready mbox.  @code{b4} is
responsible for downloading the thread, so the caller must point b4's
configuration option @code{b4.midmask} to the appropriate public-inbox
URL.

@item b
@itemx M-x piem-b4-am-ready-from-mbox
@findex piem-b4-am-ready-from-mbox
Like @code{piem-b4-am-ready-from-mid}, but process a local mbox rather
than identifying the thread based on the specified message ID.

@end table

@node Applying patches without a public-inbox archive
@section Applying patches without a public-inbox archive

Much of the functionality described in the previous sections can work
even if messages aren't available in a public-inbox archive.
@code{piem-am} and @code{piem-b4-am-from-mid} try to generate the
am-ready mbox from a local source (e.g., via Notmuch or Gnus) before
falling back to downloading the thread from a public-inbox archive.

@cindex mailscripts
Also, for those not working with public-inbox archives, it's worth
checking out @url{https://git.spwhitton.name/mailscripts/,mailscripts},
a nice set of Debian-focused tools by Sean Whitton that provides, among
other things, functionality for applying patch series, including
b4-inspired patch extraction.

@findex piem-notmuch-extract-patch-am-ready-mbox
@vindex piem-am-ready-mbox-functions
If you would like to use piem but prefer to use mailscripts'
@code{notmuch-extract-patch} script rather than b4 to prepare an
am-ready mbox, you can add the
@code{piem-notmuch-extract-patch-am-ready-mbox} function to
@code{piem-am-ready-mbox-functions}.


@node Miscellaneous functionality
@chapter Miscellaneous functionality

@node Injecting messages into a Maildir directory
@section Injecting messages into a Maildir directory
@cindex Maildir

public-inbox allows you to follow lists through several mechanisms
(@pxref{public-inbox}).  You may prefer different methods for different
projects depending on things like how actively you are following the
development and how high traffic the list is.  For a project you
maintain, perhaps you want to receive every message as regular mail.
For a project you actively follow and occasionally contribute to, you
may prefer to not clutter your local mail store and instead follow via
read-only NNTP or IMAP in Gnus (which may or may not be your MUA).  And
for a project you're new to or are digging into for a particular reason,
HTTP via EWW may be all you need.

@findex piem-inject-thread-into-maildir
@vindex piem-maildir-directory
Depending on your mail setup, a problem with this approach is that it
can be inconvenient to start participating in a thread that you aren't
reading in your regular MUA (e.g., if you use notmuch.el to read your
regular mail but are following a project via NNTP in Gnus).  In this
case, you can use the command @code{piem-inject-thread-into-maildir}
to move the thread's messages into a local Maildir directory specified
by the current inbox's @code{:maildir} value in @code{piem-inboxes},
falling back to @code{piem-maildir-directory}.  By default the command
downloads the entire thread for the message ID associated with the
current buffer.  A prefix argument restricts the download to only the
message.

@vindex piem-after-mail-injection-functions
After the messages are injected, each function in
@code{piem-after-mail-injection-functions} is called with the message ID
that was used to identify the thread.  This can be used to pop to the
message in your mail client.  For example, Notmuch users may want
something like this:

@lisp
(defun my/notmuch-new-and-show (mid)
  (message "Running notmuch new")
  (call-process notmuch-command nil nil nil "new")
  (notmuch-show (concat "id:"  mid)))

(add-hook 'piem-after-mail-injection-functions
          #'my/notmuch-new-and-show)
@end lisp

@vindex piem-mail-injection-skipif-predicate
@findex piem-notmuch-known-mid-p
To prevent duplicate messages from being written on subsequent calls to
@code{piem-inject-thread-into-maildir}, you can set
@code{piem-mail-injection-skipif-predicate} to a function that returns
non-nil if a message ID is known and should be skipped.  For Notmuch,
@code{piem-notmuch} provides a function that works for this purpose,
@code{piem-notmuch-known-mid-p}:

@lisp
(setq piem-mail-injection-skipif-predicate
      #'piem-notmuch-known-mid-p)
@end lisp

@node Copying public-inbox URLs
@section Copying public-inbox URLs

@findex piem-copy-mid-url
When referring to a message from a public-inbox archive, a common format
to use is a URL that points to a specific archive and ends with
@code{/$INBOX/$MESSAGE_ID}, e.g.,
@url{https://public-inbox.org/meta/20190108015420.GA28903@@dcvr}.
Calling @code{piem-copy-mid-url} (available in the @code{piem-dispatch}
transient) constructs such a URL, using the message ID and inbox
asscociated with the current buffer, and then copies the URL to the kill
ring.  When a prefix agument is given, @code{browse-url} is called after
copying the URL.

@vindex piem-browse-url-browser-function
Note that EWW works nicely with public-inbox's HTTP interface.  If you'd
prefer it to be invoked even though it's not your default browser (as
configured by @code{browse-url-browser-function}), you can set
@code{piem-browse-url-browser-function} to @code{eww-browse-url}.

@findex piem-notmuch-mode
@findex piem-notmuch-show-get-public-inbox-link
For notmuch.el users, there's an additional entry point for copying
public-inbox URLs: enabling @code{piem-notmuch-mode} adds a ``piem''
candidate to archives offered by
@code{notmuch-show-stash-mlarchive-link} and
@code{notmuch-show-stash-mlarchive-link-and-go}.


@node Contributing
@chapter Contributing

Patches, bug reports, and other feedback are welcome.  Please send a
plain-text email to @email{piem@@inbox.kyleam.com}.  Messages that
include this address are public and available as public-inbox archives
at @url{https://inbox.kyleam.com/piem}.  Note that this is not a mailing
list.  Updates can be followed through one of public-inbox's pull
methods (@pxref{public-inbox}).  This means it is particularly important
to @emph{not} drop participants when replying.

You can, unsurprisingly, use piem to work on piem by adding an entry
like this to @code{piem-inboxes}.

@lisp
("piem"
 :coderepo "<path/to/local/clone>"
 :address "piem@@inbox.kyleam.com"
 :url "https://inbox.kyleam.com/piem/")
@end lisp

The source repository is available at @url{https://git.kyleam.com/piem}.
Here are some guidelines for sending patches:

@itemize
@item Please send patches inline rather than as attachments.

If you're using @code{git send-email}, you may want to set
@code{sendemail.to} to @code{piem@@inbox.kyleam.com} in your local
repository.

@item Specify the base commit.

This can be done via the @code{--base=} option of @code{git
format-patch} or by configuring @code{format.useAutoBase}.

@item Keep rerolls in the same thread.

In general, prefer to keep iterations of a patch series in the same
thread, labeling rerolls with an appropriate version.

@item Consider adding a range-diff to the cover letter of rerolls.

For a multi-patch series, @code{git range-diff} can often provide a nice
overview of the changes between the previous and last iteration.  Note
that @code{git format-patch} has a @code{--range-diff} convenience
option.

@end itemize


@node Related projects and tools
@chapter Related projects and tools

Here's a (short and incomplete) list of tools that you, as someone that
cared to look at piem's manual, may be interested in---things that
either have some connection to public-inbox or to using an email-based
development workflow in Emacs.  Corrections and additions are welcome.

@itemize

@item
git-email provides an Emacs interface for preparing and sending patches.

@url{https://sr.ht/~yoctocell/git-email/}

@cindex magit
@item
@code{git range-diff} is a built-in Git command for comparing iterations
of a patch series.  If you use Magit, the
@url{https://github.com/magit/magit-tbdiff,magit-tbdiff} extension
provides an interface for @code{git range-diff}.

@url{https://kernel.org/pub/software/scm/git/docs/git-range-diff.html}

@item
grokmirror enables efficient replication of large Git repository
collections.  The HTTP interface for public-inbox exposes a
grokmirror-compatible manifest.js.gz endpoint.

@url{https://git.kernel.org/pub/scm/utils/grokmirror/grokmirror.git}

@cindex mailscripts
@item
mailscripts, mentioned earlier in the manual (@pxref{Applying patches
without a public-inbox archive}), is a ``collection of scripts for
manipulating e-mail on Debian'', including b4-inspired patch extraction
and an Emacs interface for some of the functionality.

@url{https://git.spwhitton.name/mailscripts/}

@end itemize

A list of more public-inbox-related tools is at
@url{https://public-inbox.org/clients.txt}.


@node GNU Free Documentation License
@chapter GNU Free Documentation License
@include fdl-1.3.texi

@node Key Index
@unnumbered Key Index

@printindex ky

@node Variable Index
@unnumbered Variable Index

@printindex vr

@node Lisp Function Index
@unnumbered Function Index

@printindex fn

@node Concept Index
@unnumbered Concept Index

@printindex cp


@bye