aboutsummaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2021-05-20Minor visual tweak in outputKonstantin Ryabitsev
Group patch output inside the indented ---, and all processing messages before the indent. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-20Fix a crash on incomplete/missing threadsKonstantin Ryabitsev
Properly handle situation where we can get a None as well as an empty message list. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-18Tweak output filenamesKonstantin Ryabitsev
Save am-able maildirs as .eml files, as this is more likely to be properly recognized format-wise by vim and friends. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-18Document save-maildirs and -M optionsKonstantin Ryabitsev
Update the manpage to record the new flags. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-18Don't use mboxo for anythingKonstantin Ryabitsev
While trying to figure out some odd DKIM failures, I've discovered that there is an important incompatibility between git's idea of what "mbox" format is, and Python's mboxo implementation -- at least when it comes to treating "\nFrom " escapes. According to the "original mbox" standard, when a message body contains a "\nFrom " sequence, it should be converted to "\n>From " in order not to confuse the parser. When reading messages in that format, clients are supposed to back-convert "\n>From " into their original form. This is the so-called "mboxo" format, which is what Python's mailbox.mbox supports: https://docs.python.org/3/library/mailbox.html#mailbox.mbox The "mboxrd" format was created to avoid a corruption problem whereas a body that legitimately contains "\n>From " would be wrongly converted into "\nFrom " upon parsing the mailbox, so mboxrd standard requires that, when saving a mailbox, "\n>From " sequences are additionally escaped as "\n>>From ". This is the format public-inbox supports, so when we grab mailboxes from remote, they are in mboxrd format. Git will try to guess the format of the mbox file, but it will ONLY back-convert "\n>From " sequences when you specifically tell it that it's "mboxrd" format, even when it's in fact "mboxo": git am --patch-format=mboxrd If you don't force the mboxrd format, git-am will preserve all escaped "\n>From " lines as-is. We've been previously operating on the assumption that git-am's mbox support properly implements "mboxo", but this was wrong, resulting in some commits like the following: https://git.kernel.org/torvalds/c/137733d08f4a This large-ish change ditches all internal use of Python's mboxo. When asked to save mbox files, we will save them without any escaping, the way git-am (i.e. git-mailsplit) espects them. The same goes when we're outputting to stdout. There is also a way now to pass -M to both "b4 am" and "b4 mbox" that will save things as maildirs -- git-am supports this natively and thus avoids any possible parsing ambiguities. You can set a config option b4.save-maildirs=yes to make this the default behaviour. The fallout of this is fairly benign, if annoying. There is no situation in which a patch would have "\nFrom " as part of its body, so the problem only affected commit messages. We will have a handful of these sprinkled around the trees, and will hopefully not introduce any new ones once everyone switches to the b4 version that outputs things in the format git-am expects. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-17Allow passing entire mbox via stdinKonstantin Ryabitsev
Per request, allow passing entire mbox files via stdin, allowing fully pipe-through operation from something like mutt: b4 am -sl -m - -o - Suggested-by: Peter Zijlstra <peterz@infradead.org> Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org> Link: https://lore.kernel.org/tools/YFETLu8TKWI2WlSF@hirez.programming.kicks-ass.net
2021-05-17Perform mboxo unescaping before DKIM checkKonstantin Ryabitsev
Python's mailbox will not automatically remove mboxo escaping, so perform this manually before passing the message to dkim for verification. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-17Implement partial rerollKonstantin Ryabitsev
It has been a common request to support partial series rerolls where someone sends an amended patch as a follow-up to a previous series, e.g.: [PATCH v3 1/3] Patch one [PATCH v3 2/3] Patch two \- Re: [PATCH v3 2/3] Patch two Looks good, but please fix this $small_thing \- [PATCH v4 2/3] Patch two [PATCH v3] Patch three Previously, b4 refused to consider v4 as a complete new series, but now it will properly perform a partial reroll, but only in the cases where such patches are sent as follow-ups to the exact same patch number in the previous series: [PATCH v3->v4 1/3] Patch one [PATCH v4 2/3] Patch two [PATCH v3->v4 3/3] Patch three Reported-by: Dan Williams <dan.j.williams@intel.com> Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org> Link: https://lore.kernel.org/r/CAPcyv4ggbuHbqKV33_TpE7pqxvRag34baJrX3yQe-jXOikoATQ@mail.gmail.com
2021-05-14Don't try to read stdin multiple timesKonstantin Ryabitsev
With multiple entry points for calling get_msgid(), we may end up trying to read stdin multiple times. Call it once to ensure that this doesn't happen. Suggested-by: Morten Linderud <foxboron@archlinux.org> Improved-by: Kyle Meyer <kyle@kyleam.com> Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org> Link: https://lore.kernel.org/tools/87im4fi55o.fsf@kyleam.com
2021-05-14Restore check for attestation-check-dkimKonstantin Ryabitsev
Seems we have lost this check in the rewrite, so restore it to make sure that we only check dkim if b4.attestation-check-dkim == 'yes' (default). Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-14Improve subject parsing for bracketed prefixesKonstantin Ryabitsev
Look in all of the brackets and reconstitute the subject based on what we find there. This way we properly handle even the following: Subject: [foo-list] [PATCH [RFC] v1 x/n] [RESEND] foo: do foo Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-14Ensure trailers are tracked with source messagesKonstantin Ryabitsev
When we aggregate trailers, make sure that we track their originating messages so we can properly check attestation on all of them. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-13Update patatt submodule to 0.3.0Konstantin Ryabitsev
Nothing really different in 0.3.0, just a few cleanups. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-12Fix DKIM check on headers that don't lowercase hKonstantin Ryabitsev
The h= field headers may not be lowercased, so make sure we handle that when looking if the date header is signed. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-12Properly fail with BADSIG on bad signatureKonstantin Ryabitsev
Fix logic error where we incorrectly reported "No key" when it was actually "BADSIG". Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-12Force datetime to UTC if it's nativeKonstantin Ryabitsev
We always want the datetime object to be tz-aware, but certain Date: header formats result in timezone-naive variants. For those cases, just pretend it's UTC, as that's sufficiently accurate for our purposes. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-11Fix crash when PGP key not in default keyringKonstantin Ryabitsev
Catch KeyError instead of backtracing. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-11Clean up after ourselvesKonstantin Ryabitsev
Remember to remove the temporary mbox if we're done with it. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-11Rudimentary support for showing patatt key infoKonstantin Ryabitsev
I expect that we'll have better keyring management tooling in the future, but for now show some rudimentary information about patatt keys used in a thread via --show-keys, e.g.: b4 mbox --show-keys 20210511143536.743919-1-konstantin@linuxfoundation.org b4 mbox --show-keys 20210507181322.172569-1-konstantin@linuxfoundation.org Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-11Fix for DKIM signatures without t= fieldKonstantin Ryabitsev
Many DKIM signatures just sign the Date: field and do not include the t= timestamp. Properly handle this situation when we're checking for drift. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-11Reimplement attestation-staleness-daysKonstantin Ryabitsev
Looks like we lost this feature in the rewrite, so reimplement it again. This commit also removes obsolete configuration options and sets the default attestation check level at "softfail". Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-11Python 3.6 compatibility fixesKonstantin Ryabitsev
Looks like subscripting list[] and dict[] for typing hints is not supported in python-3.6. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-11Remove sendemail-validte hookKonstantin Ryabitsev
This has moved into patatt, so no need to duplicate functionality. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-11Reimplement attestation code one more timeKonstantin Ryabitsev
Move end-to-end attestation code into its own library: patatt. See https://git.kernel.org/pub/scm/utils/patatt/patatt.git/about/ It is included into b4 as a submodule, but you will need to init it first: git submodule update --init This change significantly simplifies our attestation code, dropping thousands of lines of rather hairy code. Notably, patatt-style attestation is incompatible with previous attestation implementations done directly in b4, but that's just as well -- we've always marked it as "experimental" and the lack of adoption was proving that we weren't on the right path. Next to come is keyring management and documentation. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-03-01Loosen compatible release identifiers for install_requiresKyle Meyer
The install_requires entries use a compatible release operator. As an example, "requests~=2.24.0" maps to a requirement of ">= 2.24.0 and == 2.24.*". With the current version of requests (2.25.1), this leads to a ContextualVersionConflict failure at runtime. Allowing only Z to tick in version X.Y.Z seems unnecessarily strict unless there are known problems with a particular release, and it makes it more difficult for distributions to package b4. Drop the trailing digit from all of the version identifiers, allowing both Y and Z to increase. Signed-off-by: Kyle Meyer <kyle@kyleam.com> Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-12-28Save to/cc headers as-is for trackingKonstantin Ryabitsev
If we clean the to/cc headers to get rid of all unicode escaping, we run into a Python bug that is unable to properly parse addresses, e.g.: In [5]: from email import utils In [6]: utils.getaddresses(['foo <foo@bar.com>']) Out[6]: [('foo', 'foo@bar.com')] In [7]: utils.getaddresses(['Shuming [范書銘] <shumingf@realtek.com>']) Out[7]: [('', 'Shuming'), ('', ''), ('', '范書銘'), ('', ''), ('', 'shumingf@realtek.com')] If we store the headers as-is from the original message, we are less likely to run into this bug, as all non-ascii sequences should be qp-escaped in the original headers: =?big5?B?U2h1bWluZyBbrVOu0bvKXQ==?= <shumingf@realtek.com> This doesn't fix the underlying bug in Python, but works around it. Reported-by: Mark Brown <broonie@kernel.org> Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-12-21Add -f to "b4 mbox" to filter dupesKonstantin Ryabitsev
When saving to a maildir, add option to filter out dupes. Note, that this requires going through the entire maildir to collect message-ids, so it's not going to be a great experience on large maildirs. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-12-21Check if -o is a maildirKonstantin Ryabitsev
When -o is a valid maildir, then instead of saving an .mbox file add messages to the maildir instead. This should allow "b4 mbox" to quickly add threads to existing mail spools. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-12-16Recognize merged PRs when attempting auto-baseKonstantin Ryabitsev
When we come across a pull request that doesn't list its base (i.e. not generated using git-request-pull), we try to figure out the base ourselves by doing merge-base. However, if the pull request is already merged, then that is going to be a useless operation, so recognize this situation and bail out early. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-12-15Raise RuntimeError instead of exitingKonstantin Ryabitsev
It's a bad habit to exit anywhere other than from main() anyway. I should fix all cases of that in order to be both more pythonic and library-friendly. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-12-15Convert mboxrd to mboxoKonstantin Ryabitsev
Public-inbox emits mboxrd, but Python only understands mboxo, so we need to convert from mboxrd to mboxo before passing the retrieved results to mailbox.mbox. Reported-by: Linus Torvalds <torvalds@linux-foundation.org> Link: https://lore.kernel.org/r/CAHk-=whRm2sKHeY-YQqxEJF=d9fGhnU2ajJs9i7CKC4feuPMTA@mail.gmail.com Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-12-14Allow passing extra opts to git-format-patchKonstantin Ryabitsev
We probably want to be able to tweak the output of git-format-patch based on which list we're running it for (e.g. passing --minimal or --histogram), so make it possible to pass extra parameters to the git command. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-12-14Fix crasher when we don't use -g with b4 prKonstantin Ryabitsev
If we're not passing -g to "b4 pr -e", then we should try to see if we are inside a git checkout and use that as our source. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-12-11Rework b4 pr exploder for transparency log needsKonstantin Ryabitsev
Two services we'll be running in the near future: 1. Transparency log for all pull requests 2. Auto-exploder for pull requests that can send auto-exploded patches to all the same recipients. This requires quite a bit more testing and refinement, but the core of the functionality is there. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-12-10DKIM: remove extra public-inbox headersKonstantin Ryabitsev
The reason alsa-devel DKIM verification is failing is because the List-Archive header is included in the hashed value. This header is added by public-inbox to all messages retrieved via the API, so try ejecting those headers and retrying verification. Link: https://public-inbox.org/meta/20201210202145.7agtcmrtl5jec42d@chatter.i7.local Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-12-09Only check allow-list of trailers in follow-upsKonstantin Ryabitsev
We only need to check against the list of known non-person trailers if we're looking at follow-up messages. Any trailers we see in the actual commit messages can be taken at their face value. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-12-09Add BugLink to non-person trailersKonstantin Ryabitsev
Looks like BugLink: is a trailer used by Intel. Reported-by: Linus Walleij <linus.walleij@linaro.org> Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-12-07Up version to 0.7.0-devKonstantin Ryabitsev
Open the development round for 0.7. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-12-07Update version to 0.6.1 as hotfixv0.6.1Konstantin Ryabitsev
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-12-07Add dkimpy and dnspython to install_requiresKonstantin Ryabitsev
DKIM verification is a very useful feature for b4, so let's make it a requirement for anything installed from pip, since it's a simple enough operation. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-12-07Don't crash if dkimpy is not installedKonstantin Ryabitsev
Something I should have found out before I tagged 0.6.0. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-12-07Prepare for 0.6.0 releasev0.6.0Konstantin Ryabitsev
I think it's time to unleash this on the wider audience. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-12-07Try all DKIM headers if failed on the firstKonstantin Ryabitsev
dkim.verify will only try the topmost DKIM-Signature header, so in case of a failure, pop the failed header and retry with the next one (if any). Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-12-07Silence some pycharm warningsKonstantin Ryabitsev
PyCharm is warning that the list item can be None, but we already check for that, so silence the warning. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-12-07Fix logic when adding trailers without DKIM sigsKonstantin Ryabitsev
If the message with a follow-up trailer did not include a DKIM signature, we didn't show it in the report for added trailers (we were still adding it to the resulting message). Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-12-04Limit slug length by 100 charactersKonstantin Ryabitsev
Some subjects are still too long and hit FS file length limits. Since they are supposed to be human-friendly anyway, limit them by 100 characters. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-12-03Fix ReST in README.rstKonstantin Ryabitsev
Looks like it's not properly rendering on git.kernel.org. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-12-02Move the g= hash back into the XPH headerKonstantin Ryabitsev
I didn't like it as a separate header, so move it back into the X-Patch-Hashes header, but only add it if it's present. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-12-02Fix logic on git-patch-id hash generationKonstantin Ryabitsev
We're no longer returning here, so we need to flip our logic around. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-12-02Don't give up on attestation if there's no indexKonstantin Ryabitsev
Patches created with quilt will have no indexes, so git patch-id is refusing to generate a hash for them (somehow, though why?). At any rate, don't give up on attesting these patches even without the git's patch-id. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Generated by cgit. Colors based on the Modus themes.