aboutsummaryrefslogtreecommitdiff
path: root/b4/mbox.py
AgeCommit message (Collapse)Author
2021-09-29shazam: implement better merging stringsKonstantin Ryabitsev
b4 shazam will now: 1. give you a better default message for merging FETCH_HEAD 2. reuse the conver letter for the merge message, giving you the example invocation for using it Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-09-29shazam: allow configuring "git am" flagsKonstantin Ryabitsev
Setting b4.git-am-flags should allow you to specify what additional flags to pass to "git am" before shazaming a bunch of patches on to the current branch. Example entry in ~/.gitconfig [b4] git-am-flags = -s Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-09-24am/shazam: Restore thank-you recordingKees Cook
After the shazam refactoring, the thank-you recording went missing from both the "b4 shazam -A" and "b4 am" workflows. Restore the calls. Signed-off-by: Kees Cook <keescook@chromium.org> Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-09-21Simplify sparse worktree preparationKonstantin Ryabitsev
Looks like we don't need to have the files in place to run "git am", so skip that step and just init the sparsest possible tree. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-09-21Add "b4 shazam" that is like b4 am + git amKonstantin Ryabitsev
By popular demand, provide a way to apply series straight to a git repository. By default, we're still going the safest possible route: - create a sparse worktree consisting just of the files being modified - run "git am" against the temporary worktree - if "git am" went well, fetch from the temporary worktree into our current tree and leave everything in FETCH_HEAD - unless we're running "b4 shazam -A" in which case we just apply to the current HEAD (exact equivalent of b4 am -o- | git am) Further changes to come based on feedback. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-08-18Remove kludges that were needed before /all/Konstantin Ryabitsev
Extindex allows us to get rid of a couple of kludges: - we no longer need to manually backfill, as /all/ contains all sources - we can just query /all/ for new series Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-08-17Filter duplicate results when querying new seriesKonstantin Ryabitsev
When querying against /all/, we may get multiple hits for the same subject, so deal with it early. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-08-03Sanitize msgid before using it as savenameKonstantin Ryabitsev
Before using the msgid as part of the filename, clean it up to only contains sensible characters. Reported-by: Rob Herring <robherring2@gmail.com> Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-06-22Fix wrong statement on patches without indexesKonstantin Ryabitsev
Some patches don't have index information, so don't say they "apply clean to current tree" when they don't. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-06-22Tweak --guess-base outputKonstantin Ryabitsev
Run the check against current tree even if no --guess-base is specified (it's cheap). Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-06-22Additional --guess-base refinementsKonstantin Ryabitsev
Use --all by default, instead of limiting ourselves just to the current HEAD. This is actually a faster operation, because we don't have to pre-filter results. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-06-21Reimplement --guess-baseKonstantin Ryabitsev
Based on some feedback, attempt to reimplement --guess-base by looking at the file index hashes and using --find-object to locate when they were last changed. We limit this using --since and --until, so that we aren't trying to look through the entire history of the repo. For the --until date, we take the date of the patch. For the --since date, we take the timedelta using the number of days specified by --guess-lookback (default is 14 days). Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-06-11Save mbox files with proper unixfromKonstantin Ryabitsev
In order to avoid some of the more obscure charset encoding problems, we switched to using as_string() for generating messages before saving them in an mbox file. However, this uncovered a bug where the unixfrom was not actually generated and saved, despite as_bytes() and as_string() supposedly behaving identically. See: https://docs.python.org/3/library/email.message.html#email.message.EmailMessage.as_string This commit fixes the problem by properly setting the unixfrom and using the recommended (and hopefully less buggy) email.generator interface when saving mailboxes. Reported-by: Geert Uytterhoeven <geert@linux-m68k.org> Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-06-09Fix for recent function API changesKonstantin Ryabitsev
Returning a simple list of messages from get_am_ready broke a few other auxiliary functions invoked when run with -o- or -Q. Reported-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-06-07Save exploded pull requests as maildirs as wellKonstantin Ryabitsev
This moves maildir saving code into __init__.py so that we can benefit from it via other subcommands, such as pr. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-06-03Account for in-body headers when trimming bodyKonstantin Ryabitsev
When we discover that a message can only be attested after we trim the body, we *must* set the body to that version, otherwise an attacker could append arbitrary content past the l= value boundary. We already do this in the current form, but we weren't properly handing in-body headers like From: and Subject: that are used to indicate to git the patch author vs. committer. This patch set fixes that and also streamlines a few other places where we were already relying on git mailinfo calls. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-25Don't depend on List-Archive lore headerKonstantin Ryabitsev
The newer version of public-inbox is not injecting its own List-Archive headers, so stop relying on it for any purpose. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-25Avoid type error when local mbox lacks specified message IDKyle Meyer
Calling b4-am with --use-local=MBOX is supposed to abort with "Could not find MID in MBOX" if the mbox doesn't contain the specified message ID. As of 4950093c0 (Don't use mboxo for anything, 2021-05-18), a type error is signaled because get_strict_thread() returns None when there are no matches, and get_msgs() feeds this result to len(). Update get_msgs() to instead check whether the returned value evaluates to false. Signed-off-by: Kyle Meyer <kyle@kyleam.com> Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org> Link: https://lore.kernel.org/r/20210523025812.26456-1-kyle@kyleam.com
2021-05-21Return early if no messages retrievedKonstantin Ryabitsev
If we haven't been able to retrieve any messages, then exit early. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-21Move --show-keys into its own kr subcommandKonstantin Ryabitsev
There will be more keyring operations supported in the future, probably, but for now just move the --show-keys subcommand from "mbox" to "kr". Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-21Handle partial reroll of series without coverKonstantin Ryabitsev
A series may not have a cover letter, so properly handle that situation. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-21Don't crash on absent cover letterKonstantin Ryabitsev
We may not have an lmsg[0], so check for that. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-21Warn when we find an "Obsoleted-by" trailerKonstantin Ryabitsev
If we come across an obsoleted-by trailer, and we're not running with --checknewer, then output a warning that there is a newer revision. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-20Reimplement single-msgid cherrypickingKonstantin Ryabitsev
When processing -P_, filter by that msgid (and its follow-ups) early on, instead of parsing the entire thread and only then looking for matches. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-20Initial support for Obsoleted-by: trailerKonstantin Ryabitsev
Per discussion on the users list, add initial support for the "Obsoleted-by" trailer that points at the new revision for the series instead of doing a blind match by subject+from. Probably buggy and needs better support for series number collisions (right now we don't check if the newly retrieved series has a revision number greater than the revision we already have). Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-20Cherrypick from the correct series revisionKonstantin Ryabitsev
When cherrypicking by msgid and with multiple revisions available, make sure that we pick the series revision that actually contains the msgid being cherrypicked. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-20Write maildir atomicallyKonstantin Ryabitsev
It probably doesn't matter for b4 usage, but the maildir standard requires that files are written to tmp first and then moved into new (or hardlinked, really, then removed from tmp). Since nothing is reading the dir we're writing to, it's not as important to fully follow the standard when it comes to hardlinking, but let's at least move them into place once writing is completed. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-20Minor visual tweak in outputKonstantin Ryabitsev
Group patch output inside the indented ---, and all processing messages before the indent. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-20Fix a crash on incomplete/missing threadsKonstantin Ryabitsev
Properly handle situation where we can get a None as well as an empty message list. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-18Tweak output filenamesKonstantin Ryabitsev
Save am-able maildirs as .eml files, as this is more likely to be properly recognized format-wise by vim and friends. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-18Don't use mboxo for anythingKonstantin Ryabitsev
While trying to figure out some odd DKIM failures, I've discovered that there is an important incompatibility between git's idea of what "mbox" format is, and Python's mboxo implementation -- at least when it comes to treating "\nFrom " escapes. According to the "original mbox" standard, when a message body contains a "\nFrom " sequence, it should be converted to "\n>From " in order not to confuse the parser. When reading messages in that format, clients are supposed to back-convert "\n>From " into their original form. This is the so-called "mboxo" format, which is what Python's mailbox.mbox supports: https://docs.python.org/3/library/mailbox.html#mailbox.mbox The "mboxrd" format was created to avoid a corruption problem whereas a body that legitimately contains "\n>From " would be wrongly converted into "\nFrom " upon parsing the mailbox, so mboxrd standard requires that, when saving a mailbox, "\n>From " sequences are additionally escaped as "\n>>From ". This is the format public-inbox supports, so when we grab mailboxes from remote, they are in mboxrd format. Git will try to guess the format of the mbox file, but it will ONLY back-convert "\n>From " sequences when you specifically tell it that it's "mboxrd" format, even when it's in fact "mboxo": git am --patch-format=mboxrd If you don't force the mboxrd format, git-am will preserve all escaped "\n>From " lines as-is. We've been previously operating on the assumption that git-am's mbox support properly implements "mboxo", but this was wrong, resulting in some commits like the following: https://git.kernel.org/torvalds/c/137733d08f4a This large-ish change ditches all internal use of Python's mboxo. When asked to save mbox files, we will save them without any escaping, the way git-am (i.e. git-mailsplit) espects them. The same goes when we're outputting to stdout. There is also a way now to pass -M to both "b4 am" and "b4 mbox" that will save things as maildirs -- git-am supports this natively and thus avoids any possible parsing ambiguities. You can set a config option b4.save-maildirs=yes to make this the default behaviour. The fallout of this is fairly benign, if annoying. There is no situation in which a patch would have "\nFrom " as part of its body, so the problem only affected commit messages. We will have a handful of these sprinkled around the trees, and will hopefully not introduce any new ones once everyone switches to the b4 version that outputs things in the format git-am expects. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-17Allow passing entire mbox via stdinKonstantin Ryabitsev
Per request, allow passing entire mbox files via stdin, allowing fully pipe-through operation from something like mutt: b4 am -sl -m - -o - Suggested-by: Peter Zijlstra <peterz@infradead.org> Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org> Link: https://lore.kernel.org/tools/YFETLu8TKWI2WlSF@hirez.programming.kicks-ass.net
2021-05-17Implement partial rerollKonstantin Ryabitsev
It has been a common request to support partial series rerolls where someone sends an amended patch as a follow-up to a previous series, e.g.: [PATCH v3 1/3] Patch one [PATCH v3 2/3] Patch two \- Re: [PATCH v3 2/3] Patch two Looks good, but please fix this $small_thing \- [PATCH v4 2/3] Patch two [PATCH v3] Patch three Previously, b4 refused to consider v4 as a complete new series, but now it will properly perform a partial reroll, but only in the cases where such patches are sent as follow-ups to the exact same patch number in the previous series: [PATCH v3->v4 1/3] Patch one [PATCH v4 2/3] Patch two [PATCH v3->v4 3/3] Patch three Reported-by: Dan Williams <dan.j.williams@intel.com> Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org> Link: https://lore.kernel.org/r/CAPcyv4ggbuHbqKV33_TpE7pqxvRag34baJrX3yQe-jXOikoATQ@mail.gmail.com
2021-05-14Don't try to read stdin multiple timesKonstantin Ryabitsev
With multiple entry points for calling get_msgid(), we may end up trying to read stdin multiple times. Call it once to ensure that this doesn't happen. Suggested-by: Morten Linderud <foxboron@archlinux.org> Improved-by: Kyle Meyer <kyle@kyleam.com> Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org> Link: https://lore.kernel.org/tools/87im4fi55o.fsf@kyleam.com
2021-05-11Fix crash when PGP key not in default keyringKonstantin Ryabitsev
Catch KeyError instead of backtracing. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-11Clean up after ourselvesKonstantin Ryabitsev
Remember to remove the temporary mbox if we're done with it. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-11Rudimentary support for showing patatt key infoKonstantin Ryabitsev
I expect that we'll have better keyring management tooling in the future, but for now show some rudimentary information about patatt keys used in a thread via --show-keys, e.g.: b4 mbox --show-keys 20210511143536.743919-1-konstantin@linuxfoundation.org b4 mbox --show-keys 20210507181322.172569-1-konstantin@linuxfoundation.org Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-11Reimplement attestation code one more timeKonstantin Ryabitsev
Move end-to-end attestation code into its own library: patatt. See https://git.kernel.org/pub/scm/utils/patatt/patatt.git/about/ It is included into b4 as a submodule, but you will need to init it first: git submodule update --init This change significantly simplifies our attestation code, dropping thousands of lines of rather hairy code. Notably, patatt-style attestation is incompatible with previous attestation implementations done directly in b4, but that's just as well -- we've always marked it as "experimental" and the lack of adoption was proving that we weren't on the right path. Next to come is keyring management and documentation. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-12-28Save to/cc headers as-is for trackingKonstantin Ryabitsev
If we clean the to/cc headers to get rid of all unicode escaping, we run into a Python bug that is unable to properly parse addresses, e.g.: In [5]: from email import utils In [6]: utils.getaddresses(['foo <foo@bar.com>']) Out[6]: [('foo', 'foo@bar.com')] In [7]: utils.getaddresses(['Shuming [范書銘] <shumingf@realtek.com>']) Out[7]: [('', 'Shuming'), ('', ''), ('', '范書銘'), ('', ''), ('', 'shumingf@realtek.com')] If we store the headers as-is from the original message, we are less likely to run into this bug, as all non-ascii sequences should be qp-escaped in the original headers: =?big5?B?U2h1bWluZyBbrVOu0bvKXQ==?= <shumingf@realtek.com> This doesn't fix the underlying bug in Python, but works around it. Reported-by: Mark Brown <broonie@kernel.org> Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-12-21Add -f to "b4 mbox" to filter dupesKonstantin Ryabitsev
When saving to a maildir, add option to filter out dupes. Note, that this requires going through the entire maildir to collect message-ids, so it's not going to be a great experience on large maildirs. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-12-21Check if -o is a maildirKonstantin Ryabitsev
When -o is a valid maildir, then instead of saving an .mbox file add messages to the maildir instead. This should allow "b4 mbox" to quickly add threads to existing mail spools. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-12-14Fix crasher when we don't use -g with b4 prKonstantin Ryabitsev
If we're not passing -g to "b4 pr -e", then we should try to see if we are inside a git checkout and use that as our source. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-12-01Deal with dupe follow-up trailers sent to coverKonstantin Ryabitsev
Since we now include the message object into the followup-trailer list (for DKIM verification purposes), we no longer auto-dedupe duplicate trailers. Add some extra logic to handle that. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-11-30Properly clean up temporary threadfileKonstantin Ryabitsev
Now that we always create a strict-threaded file, always clean it up after we are done. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-11-30If using -m, reduce local mbox to strict threadKonstantin Ryabitsev
When passing a local mbox, don't assume that it is going to contain a strict thread already -- it can be just a local mailbox via something like mbsync. This grabs actual thread from the mailbox before looking at individual messages. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-11-25Unbreak thanks-trackingKonstantin Ryabitsev
Don't forget to increment where we are when doing "continue". Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-11-24Add --cc-trailers option to b4 amKonstantin Ryabitsev
By request, add ability to copy all addresses from the email's "Cc" header into Cc: trailers, unless they are already mentioned in some other trailer. Requested-by: Arnaldo Carvalho de Melo <acme@kernel.org> Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-11-24Fix crash on incomplete series thanks trackingKonstantin Ryabitsev
Now that we do our best to track incomplete series, don't crash when we come across one. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-11-24Improve ty with cherrypicked subsetsKonstantin Ryabitsev
Record patch counters when we start tracking series so we properly indicate in the thank-you note which ones got applied. Additionally, indicate in the subject when we're reporting on a subset of a larger series. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2020-10-27Fix a crasher discovered in testingKonstantin Ryabitsev
Don't crash on incomplete threads when trying to carry over previous trailers. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Generated by cgit. Colors based on the Modus themes.