aboutsummaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2021-08-17Move dedupe code into central locationKonstantin Ryabitsev
We want to dedupe all threads we retrieve from public-inbox, so do this in the central place instead of only when doing get_strict_tread(). Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-08-17Reimplement deduping based on list-id preferenceKonstantin Ryabitsev
Instead of relying on DKIM validation, use list-id preference when dealing with multiple messages matching the same message-id. We may end up adding an attestation check to it as well in the future. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-08-16When deduping, prefer DKIM-validating messagesKonstantin Ryabitsev
With newer lore.kernel.org and /all/, we get duplicate messages when message bodies are different due to one of the messages passing through a DKIM-compliant list, and another one through something that injects in-body or in-subject junk. When dealing with duplicates, check both for DKIM status and prefer the message that actually passes DKIM validation. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-08-05Handle decoding incorrectly encoded headersKonstantin Ryabitsev
Sometimes the encoding indicated in the header lies and it's not actualy that codepage at all. When that happens, just replace errors and continue. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-08-03Parse just headers when extracting message ID from stdin mboxKyle Meyer
When the mbox and am subcommands grab a message ID from the mbox on stdin, they call message_from_bytes(), which in turn calls BytesParser().parsebytes(s). parsebytes() has a headersonly parameter that can be used to tell it to stop parsing after reading the headers. The headers are all that's needed here, so use BytesParser directly and set headersonly. Signed-off-by: Kyle Meyer <kyle@kyleam.com> Link: https://lore.kernel.org/tools/20210717164836-mutt-send-email-mst@kernel.org/ Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-08-03Avoid decoding errors when extracting message ID from stdinKyle Meyer
The mbox, am, and pr subcommands accept an mbox on stdin and extract the message ID. When stdin.read() is called, Python assumes the encoding is locale.getpreferredencoding(False). This may not match the content encoding, leading to a decoding error. Instead feed the stdin bytes to message_from_bytes(), which leads to a decode('ASCII', errors='surrogateescape') underneath. That's sufficient to get the message ID from the ASCII headers. Reported-by: Michael S. Tsirkin <mst@redhat.com> Signed-off-by: Kyle Meyer <kyle@kyleam.com> Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-08-03Don't consider signature contents for trailersKonstantin Ryabitsev
Drop anything in the body below "-- " before parsing the contents for trailers. This won't catch all possible situations, as the "-- " standard is a bit of a dying standard, so add a list of known baddies like "Phone:" and "Email:" that are likely to trip us up. Reported-by: Uwe Kleine-König <u.kleine-koenig@pengutronix.de> Link: https://lore.kernel.org/tools/20210719213535.vw3u4yg5mgxqysaf@pengutronix.de/ Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-08-03Sanitize msgid before using it as savenameKonstantin Ryabitsev
Before using the msgid as part of the filename, clean it up to only contains sensible characters. Reported-by: Rob Herring <robherring2@gmail.com> Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-06-22Fix wrong statement on patches without indexesKonstantin Ryabitsev
Some patches don't have index information, so don't say they "apply clean to current tree" when they don't. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-06-22Allow '.git' to be a file for worktreesRob Herring
With multiple git worktrees, '.git' can be a file pointing to the real '.git' directory, so the current check for a directory is too strict. Signed-off-by: Rob Herring <robh@kernel.org> Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org> Link: https://lore.kernel.org/r/20210621204335.1627303-1-robh@kernel.org
2021-06-22Tweak --guess-base outputKonstantin Ryabitsev
Run the check against current tree even if no --guess-base is specified (it's cheap). Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-06-22Additional --guess-base refinementsKonstantin Ryabitsev
Use --all by default, instead of limiting ourselves just to the current HEAD. This is actually a faster operation, because we don't have to pre-filter results. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-06-21Reimplement --guess-baseKonstantin Ryabitsev
Based on some feedback, attempt to reimplement --guess-base by looking at the file index hashes and using --find-object to locate when they were last changed. We limit this using --since and --until, so that we aren't trying to look through the entire history of the repo. For the --until date, we take the date of the patch. For the --since date, we take the timedelta using the number of days specified by --guess-lookback (default is 14 days). Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-06-17Don't append .git unnecessarilyKonstantin Ryabitsev
We already do this automatically elsewhere, so this causes a problem if we do it again. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-06-11Test to make sure mbox files contain unixfromKonstantin Ryabitsev
Start a test suite for generated mbox files. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-06-11Save mbox files with proper unixfromKonstantin Ryabitsev
In order to avoid some of the more obscure charset encoding problems, we switched to using as_string() for generating messages before saving them in an mbox file. However, this uncovered a bug where the unixfrom was not actually generated and saved, despite as_bytes() and as_string() supposedly behaving identically. See: https://docs.python.org/3/library/email.message.html#email.message.EmailMessage.as_string This commit fixes the problem by properly setting the unixfrom and using the recommended (and hopefully less buggy) email.generator interface when saving mailboxes. Reported-by: Geert Uytterhoeven <geert@linux-m68k.org> Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-06-10Start using pytest for the test frameworkKonstantin Ryabitsev
Since we're not caring about 2.x compatibility, pytest seems to be a good candidate for this job. Obviously, there's a lot of ground to cover, but the goal is to do all future modifications with tests added so we can reduce regressions. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-06-09Update patatt to latest 0.4.6Konstantin Ryabitsev
Fixes things that should have been fixed in 0.4.5. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-06-09Fix sloppy trailer handlingKonstantin Ryabitsev
When returning sloppy trailers, make sure we always return a 4-member list, which includes the provenant LoreMessage itself. Reported-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-06-09Fix for recent function API changesKonstantin Ryabitsev
Returning a simple list of messages from get_am_ready broke a few other auxiliary functions invoked when run with -o- or -Q. Reported-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-06-08Perform an encoding bug avoidance danceKonstantin Ryabitsev
This is a kludge that I'm putting in place to avoid hitting an encoding bug with Python. We really shouldn't be making a fake RFC2822 message just so we can parse it again right away, so this is a temporary measure until I refactor how it's done. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-06-08Check if the name is nullKonstantin Ryabitsev
I'm not sure if it's a good strategy to use GitHub usernames and username@github.com addresses for this purpose, as we really need to be able to reach someone when we process their submissions. However, for the time being at least don't output None. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-06-08Initial support for github pull requestsKonstantin Ryabitsev
You can now do "b4 pr https://github.com/foo/bar/pull/NNN" and have it fetched into FETCH_HEAD. However, this also works when combined with --explode, which is the main reason for this feature's existence. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-06-08Fix crasher on unsigned FETCH_HEADKonstantin Ryabitsev
If the FETCH_HEAD is not signed, then keyid is going to be None. Don't attempt to look up UIDs in such situations. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-06-08Tentative suport for sending exploded seriesKonstantin Ryabitsev
I've been working on a way to automatically convert pull requests into series, complete with mailing them out to arbitrary destinations. This would allow folks to send a pull request to a dedicated list and have it automatically converted into a well-formed series. This is a tentative implementation that relies on git-send-email to do most of the heavy lifting. I have misgivings about using git-send-email for this purpose, but it does reduce the amount of duplicated code we would have otherwise had to write, and allows us to hook into things like tocmd/cccmd, etc. For example, adding the following to your .git/config: [sendemail "autopr"] smtpserver = [your.server.here] smtpserverport = 587 smtpencryption = tls smtpuser = [your-user] smtppass = [your-pass] transferEncoding = 8bit suppressFrom = yes confirm = never validate = no tocmd = "$(git rev-parse --show-toplevel)/scripts/get_maintainer.pl --norolestats --nol" cccmd = "$(git rev-parse --show-toplevel)/scripts/get_maintainer.pl --norolestats --nom" This would allow doing the following: b4 pr -e -f "AutoPR Exploder <autopr@yourdomain.here>" -s autopr [--dry-run] The pull request will be exploded into a patch series and sent to all the proper destinations as returned by get_maintainer.pl. We construct the message headers in a way that allow regular code review and "b4 am" usage after the auto-exploded series is sent out. If testing goes well, we'll implement this as a kernel.org service and then hook a similar implementation via Gitlab/Github. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-06-07Save exploded pull requests as maildirs as wellKonstantin Ryabitsev
This moves maildir saving code into __init__.py so that we can benefit from it via other subcommands, such as pr. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-06-07Shuffle DKIM logging code around a bitKonstantin Ryabitsev
PyCharm is unhappy with PEP conformance, so shuffle things around a bit to satisfy it. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-06-07Update to latest patatt versionKonstantin Ryabitsev
Fixes "NOKEY" vs. "BADSIG" problem (again). Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-06-07Include dkim log output when -d/--debug argument is passedPaul Barker
We can pass a logger object to dkim.verify() which will be used to report internal errors and debugging info. This can be helpful when investigating DKIM verification issues but is probably not wanted during normal operation so the log level of each message is reset to DEBUG. Each message is also prefixed with 'DKIM: ' to identify its origin when debug output is enabled. Signed-off-by: Paul Barker <paul@pbarker.dev> Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org> Link: https://lore.kernel.org/r/20210607100252.8253-3-paul@pbarker.dev
2021-06-07Handle MIME encoded-word in DKIM-Signature headersPaul Barker
As recently found in patatt [1], mail gateways and archivers may mangle headers like DKIM-Signature if they are sent as an excessively long line. An example of this occuring was found when the DKIM-Signature header generated by Microsoft Office 365 collided with the header re-encoding performed by lists.sr.ht when generating mbox archive files. This encoding causes dkim.verify() to fail. The Python email.header module provides the decode_header() and make_header() functions which can be used to handle MIME encoded-word syntax or other header manglings which may occur. Fixing up the header content using these functions before calling dkim.verify() allows the verification to succeed. [1]: https://lore.kernel.org/tools/20210531140539.7630-1-paul@pbarker.dev/ Signed-off-by: Paul Barker <paul@pbarker.dev> Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org> Link: https://lore.kernel.org/r/20210607100252.8253-2-paul@pbarker.dev
2021-06-07Add my own attestation keyKonstantin Ryabitsev
Identity verified in person! Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-06-07Add attestation key 74975C81B7E66BACKonstantin Ryabitsev
Identity not validated, but key retrieved from keys.openpgp.org, which performs an email roundtrip check. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-06-03Update patatt to 0.4.5Konstantin Ryabitsev
Fixes wrong error message for keys coming from default keyring. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-06-03Account for in-body headers when trimming bodyKonstantin Ryabitsev
When we discover that a message can only be attested after we trim the body, we *must* set the body to that version, otherwise an attacker could append arbitrary content past the l= value boundary. We already do this in the current form, but we weren't properly handing in-body headers like From: and Subject: that are used to indicate to git the patch author vs. committer. This patch set fixes that and also streamlines a few other places where we were already relying on git mailinfo calls. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-06-03Fix partial reroll TUI visuals for v1->v2Konstantin Ryabitsev
Before: ✓ [PATCH v2 1/8] selftests/x86: Test signal frame XSTATE header corruption handling ✓ [PATCH v2 2/8] x86/fpu: Prevent state corruption in __fpu__restore_sig() ✓ [PATCH 3/8] x86/fpu: Invalidate FPU state after a failed XRSTOR from a user buffer ✓ [PATCH 4/8] x86/fpu: Limit xstate copy size in xstateregs_set() ✓ [PATCH v2 5/8] x86/fpu: Sanitize xstateregs_set() ✓ [PATCH 6/8] x86/fpu: Add address range checks to copy_user_to_xstate() ✓ [PATCH 7/8] x86/fpu: Clean up the fpu__clear() variants ✓ [PATCH 8/8] x86/fpu: Deduplicate copy_xxx_to_xstate() After: ✓ [PATCH v2 1/8] selftests/x86: Test signal frame XSTATE header corruption handling ✓ [PATCH v2 2/8] x86/fpu: Prevent state corruption in __fpu__restore_sig() ✓ [PATCH v1->v2 3/8] x86/fpu: Invalidate FPU state after a failed XRSTOR from a user buffer ✓ [PATCH v1->v2 4/8] x86/fpu: Limit xstate copy size in xstateregs_set() ✓ [PATCH v2 5/8] x86/fpu: Sanitize xstateregs_set() ✓ [PATCH v1->v2 6/8] x86/fpu: Add address range checks to copy_user_to_xstate() ✓ [PATCH v1->v2 7/8] x86/fpu: Clean up the fpu__clear() variants ✓ [PATCH v1->v2 8/8] x86/fpu: Deduplicate copy_xxx_to_xstate() Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-06-02Implement trim_body supportKonstantin Ryabitsev
When a message has a developer signature but is failing the signature check, rerun it again with trim_body. If that passes, we know that the signature is failing due to mailing list junk appended to the bottom of the message. In that case, automatically trim the message body so we have exactly what the developer attested and signed. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-06-02Add *.maildir to gitignoreKonstantin Ryabitsev
Now that we can save as maildirs, add them to gitignore as well. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-06-01Fix cache aging cleanup of threadsRob Herring
The cache aging for threads was not running resulting in failures to fetch new messages in threads. Fix the empty cache check which should be for no '.msgs' directories. Fixes: 4950093c0c3e ("Don't use mboxo for anything") Signed-off-by: Rob Herring <robh@kernel.org> Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org> Link: https://lore.kernel.org/r/20210601200835.940887-1-robh@kernel.org
2021-05-28Update to newer patatt versionKonstantin Ryabitsev
Fixes public key lookups for uncommitted keys. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-28Limit 'From mboxrd@z' replacement to start of messageKyle Meyer
save_git_am_mbox() replaces 'From mboxrd@z ' with 'From git@z ' to make it clear that the output format is not mboxrd. However, all occurrences in the message are replaced, corrupting patches that contain 'From mboxrd@z '. Restrict the replacement to the first line of the message. Signed-off-by: Kyle Meyer <kyle@kyleam.com> Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org> Link: https://lore.kernel.org/r/20210528042635.24959-1-kyle@kyleam.com
2021-05-26Up version to final 0.7.0v0.7.0Konstantin Ryabitsev
I think we are ready to go with the 0.7.0 release. There's always more tweaks to add, but at this point we can benefit from wider usage. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-26Add new b4 pr flagsKonstantin Ryabitsev
The -f and -l flags are mostly used for archival purposes -- they allow to convert a pull request into a mini-archive which includes relevant discussions around all of the commits involved in it. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-26Check uids on the key when using default keyringKonstantin Ryabitsev
When the signature is validated using the default keyring, run an additional check on the UIDs and show the discrepancy if the identity used in the X-Developer-Signature header is different from the UIDs we have on the key. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-25Update attestation section in the READMEKonstantin Ryabitsev
- the default attestation policy is now "softfail" - include instructions about installing the patatt submodule Better read-the-docs style documentation will be coming in 0.8. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-25Don't depend on List-Archive lore headerKonstantin Ryabitsev
The newer version of public-inbox is not injecting its own List-Archive headers, so stop relying on it for any purpose. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-25Handle situations where -f only has an emailKonstantin Ryabitsev
Normally, -f would be 'Some Service <service@example.org>', but in case it's just "service@example.org", wrap it in angle brackets properly. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-25Fix pr -l where we introduced var collisionKonstantin Ryabitsev
When we're retrieving linked messages, make sure we don't clash with the variable holding the overall mbox. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-25Bump patatt to 0.4.2Konstantin Ryabitsev
This version returns a failure early when body modification is recognized. This is especially useful if we have to shell out to gnupg for validation. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-05-25Avoid type error when local mbox lacks specified message IDKyle Meyer
Calling b4-am with --use-local=MBOX is supposed to abort with "Could not find MID in MBOX" if the mbox doesn't contain the specified message ID. As of 4950093c0 (Don't use mboxo for anything, 2021-05-18), a type error is signaled because get_strict_thread() returns None when there are no matches, and get_msgs() feeds this result to len(). Update get_msgs() to instead check whether the returned value evaluates to false. Signed-off-by: Kyle Meyer <kyle@kyleam.com> Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org> Link: https://lore.kernel.org/r/20210523025812.26456-1-kyle@kyleam.com
2021-05-21Update patatt to 0.4.1Konstantin Ryabitsev
Minor bugfix release with better error messages for installs without any keys and unconfigured git. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>