Age | Commit message (Collapse) | Author |
|
While trying to figure out some odd DKIM failures, I've discovered that
there is an important incompatibility between git's idea of what "mbox"
format is, and Python's mboxo implementation -- at least when it comes
to treating "\nFrom " escapes.
According to the "original mbox" standard, when a message body contains
a "\nFrom " sequence, it should be converted to "\n>From " in order not
to confuse the parser. When reading messages in that format, clients are
supposed to back-convert "\n>From " into their original form. This is
the so-called "mboxo" format, which is what Python's mailbox.mbox
supports:
https://docs.python.org/3/library/mailbox.html#mailbox.mbox
The "mboxrd" format was created to avoid a corruption problem whereas a
body that legitimately contains "\n>From " would be wrongly converted
into "\nFrom " upon parsing the mailbox, so mboxrd standard requires
that, when saving a mailbox, "\n>From " sequences are additionally
escaped as "\n>>From ". This is the format public-inbox supports, so
when we grab mailboxes from remote, they are in mboxrd format.
Git will try to guess the format of the mbox file, but it will ONLY
back-convert "\n>From " sequences when you specifically tell it that
it's "mboxrd" format, even when it's in fact "mboxo":
git am --patch-format=mboxrd
If you don't force the mboxrd format, git-am will preserve all escaped
"\n>From " lines as-is.
We've been previously operating on the assumption that git-am's mbox
support properly implements "mboxo", but this was wrong, resulting in
some commits like the following:
https://git.kernel.org/torvalds/c/137733d08f4a
This large-ish change ditches all internal use of Python's mboxo. When
asked to save mbox files, we will save them without any escaping, the
way git-am (i.e. git-mailsplit) espects them. The same goes when we're
outputting to stdout.
There is also a way now to pass -M to both "b4 am" and "b4 mbox" that
will save things as maildirs -- git-am supports this natively and thus
avoids any possible parsing ambiguities. You can set a config option
b4.save-maildirs=yes to make this the default behaviour.
The fallout of this is fairly benign, if annoying. There is no situation
in which a patch would have "\nFrom " as part of its body, so the
problem only affected commit messages. We will have a handful of these
sprinkled around the trees, and will hopefully not introduce any new
ones once everyone switches to the b4 version that outputs things in the
format git-am expects.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Per request, allow passing entire mbox files via stdin, allowing fully
pipe-through operation from something like mutt:
b4 am -sl -m - -o -
Suggested-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Link: https://lore.kernel.org/tools/YFETLu8TKWI2WlSF@hirez.programming.kicks-ass.net
|
|
Python's mailbox will not automatically remove mboxo escaping, so
perform this manually before passing the message to dkim for
verification.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
It has been a common request to support partial series rerolls where
someone sends an amended patch as a follow-up to a previous series,
e.g.:
[PATCH v3 1/3] Patch one
[PATCH v3 2/3] Patch two
\- Re: [PATCH v3 2/3] Patch two
Looks good, but please fix this $small_thing
\- [PATCH v4 2/3] Patch two
[PATCH v3] Patch three
Previously, b4 refused to consider v4 as a complete new series, but now
it will properly perform a partial reroll, but only in the cases where
such patches are sent as follow-ups to the exact same patch number in
the previous series:
[PATCH v3->v4 1/3] Patch one
[PATCH v4 2/3] Patch two
[PATCH v3->v4 3/3] Patch three
Reported-by: Dan Williams <dan.j.williams@intel.com>
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Link: https://lore.kernel.org/r/CAPcyv4ggbuHbqKV33_TpE7pqxvRag34baJrX3yQe-jXOikoATQ@mail.gmail.com
|
|
Seems we have lost this check in the rewrite, so restore it to make sure
that we only check dkim if b4.attestation-check-dkim == 'yes' (default).
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Look in all of the brackets and reconstitute the subject based on what
we find there. This way we properly handle even the following:
Subject: [foo-list] [PATCH [RFC] v1 x/n] [RESEND] foo: do foo
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
When we aggregate trailers, make sure that we track their originating
messages so we can properly check attestation on all of them.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
The h= field headers may not be lowercased, so make sure we handle that
when looking if the date header is signed.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Fix logic error where we incorrectly reported "No key" when it was
actually "BADSIG".
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
We always want the datetime object to be tz-aware, but certain Date:
header formats result in timezone-naive variants. For those cases, just
pretend it's UTC, as that's sufficiently accurate for our purposes.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
I expect that we'll have better keyring management tooling in the
future, but for now show some rudimentary information about patatt keys
used in a thread via --show-keys, e.g.:
b4 mbox --show-keys 20210511143536.743919-1-konstantin@linuxfoundation.org
b4 mbox --show-keys 20210507181322.172569-1-konstantin@linuxfoundation.org
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Many DKIM signatures just sign the Date: field and do not include the t=
timestamp. Properly handle this situation when we're checking for drift.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Looks like we lost this feature in the rewrite, so reimplement it again.
This commit also removes obsolete configuration options and sets the
default attestation check level at "softfail".
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Looks like subscripting list[] and dict[] for typing hints is not
supported in python-3.6.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Move end-to-end attestation code into its own library: patatt. See
https://git.kernel.org/pub/scm/utils/patatt/patatt.git/about/
It is included into b4 as a submodule, but you will need to init it
first:
git submodule update --init
This change significantly simplifies our attestation code, dropping
thousands of lines of rather hairy code. Notably, patatt-style
attestation is incompatible with previous attestation implementations
done directly in b4, but that's just as well -- we've always marked it
as "experimental" and the lack of adoption was proving that we weren't
on the right path.
Next to come is keyring management and documentation.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
If we clean the to/cc headers to get rid of all unicode escaping, we run
into a Python bug that is unable to properly parse addresses, e.g.:
In [5]: from email import utils
In [6]: utils.getaddresses(['foo <foo@bar.com>'])
Out[6]: [('foo', 'foo@bar.com')]
In [7]: utils.getaddresses(['Shuming [范書銘] <shumingf@realtek.com>'])
Out[7]:
[('', 'Shuming'),
('', ''),
('', '范書銘'),
('', ''),
('', 'shumingf@realtek.com')]
If we store the headers as-is from the original message, we are less
likely to run into this bug, as all non-ascii sequences should be
qp-escaped in the original headers:
=?big5?B?U2h1bWluZyBbrVOu0bvKXQ==?= <shumingf@realtek.com>
This doesn't fix the underlying bug in Python, but works around it.
Reported-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Public-inbox emits mboxrd, but Python only understands mboxo, so we need
to convert from mboxrd to mboxo before passing the retrieved results to
mailbox.mbox.
Reported-by: Linus Torvalds <torvalds@linux-foundation.org>
Link: https://lore.kernel.org/r/CAHk-=whRm2sKHeY-YQqxEJF=d9fGhnU2ajJs9i7CKC4feuPMTA@mail.gmail.com
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
We probably want to be able to tweak the output of git-format-patch
based on which list we're running it for (e.g. passing --minimal or
--histogram), so make it possible to pass extra parameters to the git
command.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
If we're not passing -g to "b4 pr -e", then we should try to see if we
are inside a git checkout and use that as our source.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Two services we'll be running in the near future:
1. Transparency log for all pull requests
2. Auto-exploder for pull requests that can send auto-exploded patches
to all the same recipients.
This requires quite a bit more testing and refinement, but the core of
the functionality is there.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
The reason alsa-devel DKIM verification is failing is because the
List-Archive header is included in the hashed value. This header is
added by public-inbox to all messages retrieved via the API, so try
ejecting those headers and retrying verification.
Link: https://public-inbox.org/meta/20201210202145.7agtcmrtl5jec42d@chatter.i7.local
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
We only need to check against the list of known non-person trailers if
we're looking at follow-up messages. Any trailers we see in the actual
commit messages can be taken at their face value.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Looks like BugLink: is a trailer used by Intel.
Reported-by: Linus Walleij <linus.walleij@linaro.org>
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Open the development round for 0.7.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Something I should have found out before I tagged 0.6.0.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
I think it's time to unleash this on the wider audience.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
dkim.verify will only try the topmost DKIM-Signature header, so in case
of a failure, pop the failed header and retry with the next one (if
any).
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
PyCharm is warning that the list item can be None, but we already check
for that, so silence the warning.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
If the message with a follow-up trailer did not include a DKIM
signature, we didn't show it in the report for added trailers (we were
still adding it to the resulting message).
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Some subjects are still too long and hit FS file length limits. Since
they are supposed to be human-friendly anyway, limit them by 100
characters.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
We're no longer returning here, so we need to flip our logic around.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Patches created with quilt will have no indexes, so git patch-id is
refusing to generate a hash for them (somehow, though why?). At any
rate, don't give up on attesting these patches even without the git's
patch-id.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Since we now include the message object into the followup-trailer list
(for DKIM verification purposes), we no longer auto-dedupe duplicate
trailers. Add some extra logic to handle that.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
If we don't have dnspython, then we don't have _resolver. Make sure it
exists and check if it's not None before looking for hasattr.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Make sure we always create a Date: header, and that we're not crashing
when we try to parse a message without a Date: header.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
When passing a local mbox, don't assume that it is going to contain a
strict thread already -- it can be just a local mailbox via something
like mbsync. This grabs actual thread from the mailbox before looking at
individual messages.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
We may have 3 or 4 members in the array, so don't expect always 3.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
When displaying follow-up trailers, also indicate their DKIM status.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
There doesn't seem to be much rhyme or reason for why an address would
be in "To" or "Cc", so use both headers when finding Cc: trailer
recipients.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
By request, add ability to copy all addresses from the email's "Cc"
header into Cc: trailers, unless they are already mentioned in some
other trailer.
Requested-by: Arnaldo Carvalho de Melo <acme@kernel.org>
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
If we're processing a full https URL to the message, then unquote the
message ID before we use it.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Don't use self.expected, but actual array length when preparing
attestation report.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Don't display failures if there are no attestations available.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Multiple fixes for error messages displayed in softfail and hardfail
modes.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
If we don't find a resolve() method in dnspython, just let dkimpy do its
own lookups.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Only works for x-patch-sig style attestation, as doing DKIM attestation
requires that we unignore all headers, which just junks up the view.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
If we find an older dnspython < 2.0, don't crash but let dkim figure out
how it wants to look up TXT records on its own.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Some DKIM keys may not list v=DKIM1.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Make it possible to turn off dkim verification entirely, but leave other
attestation modes enabled.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|