| Age | Commit message (Collapse) | Author |
|---|
|
Extindex allows us to get rid of a couple of kludges:
- we no longer need to manually backfill, as /all/ contains all sources
- we can just query /all/ for new series
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
When querying against /all/, we may get multiple hits for the same
subject, so deal with it early.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
We want to dedupe all threads we retrieve from public-inbox, so do this
in the central place instead of only when doing get_strict_tread().
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Instead of relying on DKIM validation, use list-id preference when
dealing with multiple messages matching the same message-id. We may
end up adding an attestation check to it as well in the future.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
With newer lore.kernel.org and /all/, we get duplicate messages when
message bodies are different due to one of the messages passing through
a DKIM-compliant list, and another one through something that injects
in-body or in-subject junk. When dealing with duplicates, check both for
DKIM status and prefer the message that actually passes DKIM validation.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Sometimes the encoding indicated in the header lies and it's not actualy
that codepage at all. When that happens, just replace errors and
continue.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
When the mbox and am subcommands grab a message ID from the mbox
on stdin, they call message_from_bytes(), which in turn calls
BytesParser().parsebytes(s).
parsebytes() has a headersonly parameter that can be used to tell it
to stop parsing after reading the headers. The headers are all that's
needed here, so use BytesParser directly and set headersonly.
Signed-off-by: Kyle Meyer <kyle@kyleam.com>
Link: https://lore.kernel.org/tools/20210717164836-mutt-send-email-mst@kernel.org/
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
The mbox, am, and pr subcommands accept an mbox on stdin and extract
the message ID. When stdin.read() is called, Python assumes the
encoding is locale.getpreferredencoding(False). This may not match
the content encoding, leading to a decoding error.
Instead feed the stdin bytes to message_from_bytes(), which leads to a
decode('ASCII', errors='surrogateescape') underneath. That's
sufficient to get the message ID from the ASCII headers.
Reported-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Kyle Meyer <kyle@kyleam.com>
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Drop anything in the body below "-- " before parsing the contents for
trailers. This won't catch all possible situations, as the "-- "
standard is a bit of a dying standard, so add a list of known baddies
like "Phone:" and "Email:" that are likely to trip us up.
Reported-by: Uwe Kleine-König <u.kleine-koenig@pengutronix.de>
Link: https://lore.kernel.org/tools/20210719213535.vw3u4yg5mgxqysaf@pengutronix.de/
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Before using the msgid as part of the filename, clean it up to only
contains sensible characters.
Reported-by: Rob Herring <robherring2@gmail.com>
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Some patches don't have index information, so don't say they "apply
clean to current tree" when they don't.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
With multiple git worktrees, '.git' can be a file pointing to the real
'.git' directory, so the current check for a directory is too strict.
Signed-off-by: Rob Herring <robh@kernel.org>
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Link: https://lore.kernel.org/r/20210621204335.1627303-1-robh@kernel.org
|
|
Run the check against current tree even if no --guess-base is specified
(it's cheap).
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Use --all by default, instead of limiting ourselves just to the current
HEAD. This is actually a faster operation, because we don't have to
pre-filter results.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Based on some feedback, attempt to reimplement --guess-base by looking
at the file index hashes and using --find-object to locate when they
were last changed. We limit this using --since and --until, so that we
aren't trying to look through the entire history of the repo. For the
--until date, we take the date of the patch. For the --since date, we
take the timedelta using the number of days specified by
--guess-lookback (default is 14 days).
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
We already do this automatically elsewhere, so this causes a problem if
we do it again.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Start a test suite for generated mbox files.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
In order to avoid some of the more obscure charset encoding problems, we
switched to using as_string() for generating messages before saving them
in an mbox file. However, this uncovered a bug where the unixfrom was
not actually generated and saved, despite as_bytes() and as_string()
supposedly behaving identically.
See:
https://docs.python.org/3/library/email.message.html#email.message.EmailMessage.as_string
This commit fixes the problem by properly setting the unixfrom and using
the recommended (and hopefully less buggy) email.generator interface
when saving mailboxes.
Reported-by: Geert Uytterhoeven <geert@linux-m68k.org>
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Since we're not caring about 2.x compatibility, pytest seems to be a
good candidate for this job. Obviously, there's a lot of ground to
cover, but the goal is to do all future modifications with tests added
so we can reduce regressions.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Fixes things that should have been fixed in 0.4.5.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
When returning sloppy trailers, make sure we always return a 4-member
list, which includes the provenant LoreMessage itself.
Reported-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Returning a simple list of messages from get_am_ready broke a few other
auxiliary functions invoked when run with -o- or -Q.
Reported-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
This is a kludge that I'm putting in place to avoid hitting an encoding
bug with Python. We really shouldn't be making a fake RFC2822 message
just so we can parse it again right away, so this is a temporary measure
until I refactor how it's done.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
I'm not sure if it's a good strategy to use GitHub usernames and
username@github.com addresses for this purpose, as we really need to be
able to reach someone when we process their submissions. However, for
the time being at least don't output None.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
You can now do "b4 pr https://github.com/foo/bar/pull/NNN" and have it
fetched into FETCH_HEAD. However, this also works when combined with
--explode, which is the main reason for this feature's existence.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
If the FETCH_HEAD is not signed, then keyid is going to be None. Don't
attempt to look up UIDs in such situations.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
I've been working on a way to automatically convert pull requests into
series, complete with mailing them out to arbitrary destinations. This
would allow folks to send a pull request to a dedicated list and have it
automatically converted into a well-formed series.
This is a tentative implementation that relies on git-send-email to do
most of the heavy lifting. I have misgivings about using git-send-email
for this purpose, but it does reduce the amount of duplicated code we
would have otherwise had to write, and allows us to hook into things
like tocmd/cccmd, etc.
For example, adding the following to your .git/config:
[sendemail "autopr"]
smtpserver = [your.server.here]
smtpserverport = 587
smtpencryption = tls
smtpuser = [your-user]
smtppass = [your-pass]
transferEncoding = 8bit
suppressFrom = yes
confirm = never
validate = no
tocmd = "$(git rev-parse --show-toplevel)/scripts/get_maintainer.pl --norolestats --nol"
cccmd = "$(git rev-parse --show-toplevel)/scripts/get_maintainer.pl --norolestats --nom"
This would allow doing the following:
b4 pr -e -f "AutoPR Exploder <autopr@yourdomain.here>" -s autopr [--dry-run]
The pull request will be exploded into a patch series and sent to all
the proper destinations as returned by get_maintainer.pl. We construct
the message headers in a way that allow regular code review and "b4 am"
usage after the auto-exploded series is sent out.
If testing goes well, we'll implement this as a kernel.org service and
then hook a similar implementation via Gitlab/Github.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
This moves maildir saving code into __init__.py so that we can benefit
from it via other subcommands, such as pr.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
PyCharm is unhappy with PEP conformance, so shuffle things around a bit
to satisfy it.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Fixes "NOKEY" vs. "BADSIG" problem (again).
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
We can pass a logger object to dkim.verify() which will be used to
report internal errors and debugging info. This can be helpful when
investigating DKIM verification issues but is probably not wanted during
normal operation so the log level of each message is reset to DEBUG.
Each message is also prefixed with 'DKIM: ' to identify its origin when
debug output is enabled.
Signed-off-by: Paul Barker <paul@pbarker.dev>
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Link: https://lore.kernel.org/r/20210607100252.8253-3-paul@pbarker.dev
|
|
As recently found in patatt [1], mail gateways and archivers may mangle
headers like DKIM-Signature if they are sent as an excessively long
line. An example of this occuring was found when the DKIM-Signature
header generated by Microsoft Office 365 collided with the
header re-encoding performed by lists.sr.ht when generating mbox
archive files. This encoding causes dkim.verify() to fail.
The Python email.header module provides the decode_header() and
make_header() functions which can be used to handle MIME encoded-word
syntax or other header manglings which may occur. Fixing up the header
content using these functions before calling dkim.verify() allows the
verification to succeed.
[1]: https://lore.kernel.org/tools/20210531140539.7630-1-paul@pbarker.dev/
Signed-off-by: Paul Barker <paul@pbarker.dev>
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Link: https://lore.kernel.org/r/20210607100252.8253-2-paul@pbarker.dev
|
|
Identity verified in person!
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Identity not validated, but key retrieved from keys.openpgp.org, which
performs an email roundtrip check.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Fixes wrong error message for keys coming from default keyring.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
When we discover that a message can only be attested after we trim the
body, we *must* set the body to that version, otherwise an attacker
could append arbitrary content past the l= value boundary. We already do
this in the current form, but we weren't properly handing in-body
headers like From: and Subject: that are used to indicate to git the
patch author vs. committer.
This patch set fixes that and also streamlines a few other places where
we were already relying on git mailinfo calls.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Before:
✓ [PATCH v2 1/8] selftests/x86: Test signal frame XSTATE header corruption handling
✓ [PATCH v2 2/8] x86/fpu: Prevent state corruption in __fpu__restore_sig()
✓ [PATCH 3/8] x86/fpu: Invalidate FPU state after a failed XRSTOR from a user buffer
✓ [PATCH 4/8] x86/fpu: Limit xstate copy size in xstateregs_set()
✓ [PATCH v2 5/8] x86/fpu: Sanitize xstateregs_set()
✓ [PATCH 6/8] x86/fpu: Add address range checks to copy_user_to_xstate()
✓ [PATCH 7/8] x86/fpu: Clean up the fpu__clear() variants
✓ [PATCH 8/8] x86/fpu: Deduplicate copy_xxx_to_xstate()
After:
✓ [PATCH v2 1/8] selftests/x86: Test signal frame XSTATE header corruption handling
✓ [PATCH v2 2/8] x86/fpu: Prevent state corruption in __fpu__restore_sig()
✓ [PATCH v1->v2 3/8] x86/fpu: Invalidate FPU state after a failed XRSTOR from a user buffer
✓ [PATCH v1->v2 4/8] x86/fpu: Limit xstate copy size in xstateregs_set()
✓ [PATCH v2 5/8] x86/fpu: Sanitize xstateregs_set()
✓ [PATCH v1->v2 6/8] x86/fpu: Add address range checks to copy_user_to_xstate()
✓ [PATCH v1->v2 7/8] x86/fpu: Clean up the fpu__clear() variants
✓ [PATCH v1->v2 8/8] x86/fpu: Deduplicate copy_xxx_to_xstate()
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
When a message has a developer signature but is failing the signature
check, rerun it again with trim_body. If that passes, we know that the
signature is failing due to mailing list junk appended to the bottom of
the message. In that case, automatically trim the message body so we
have exactly what the developer attested and signed.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Now that we can save as maildirs, add them to gitignore as well.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
The cache aging for threads was not running resulting in failures to
fetch new messages in threads. Fix the empty cache check which should
be for no '.msgs' directories.
Fixes: 4950093c0c3e ("Don't use mboxo for anything")
Signed-off-by: Rob Herring <robh@kernel.org>
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Link: https://lore.kernel.org/r/20210601200835.940887-1-robh@kernel.org
|
|
Fixes public key lookups for uncommitted keys.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
save_git_am_mbox() replaces 'From mboxrd@z ' with 'From git@z ' to
make it clear that the output format is not mboxrd. However, all
occurrences in the message are replaced, corrupting patches that
contain 'From mboxrd@z '. Restrict the replacement to the first line
of the message.
Signed-off-by: Kyle Meyer <kyle@kyleam.com>
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Link: https://lore.kernel.org/r/20210528042635.24959-1-kyle@kyleam.com
|
|
I think we are ready to go with the 0.7.0 release. There's always more
tweaks to add, but at this point we can benefit from wider usage.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
The -f and -l flags are mostly used for archival purposes -- they allow
to convert a pull request into a mini-archive which includes relevant
discussions around all of the commits involved in it.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
When the signature is validated using the default keyring, run an
additional check on the UIDs and show the discrepancy if the identity
used in the X-Developer-Signature header is different from the UIDs we
have on the key.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
- the default attestation policy is now "softfail"
- include instructions about installing the patatt submodule
Better read-the-docs style documentation will be coming in 0.8.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
The newer version of public-inbox is not injecting its own List-Archive
headers, so stop relying on it for any purpose.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Normally, -f would be 'Some Service <service@example.org>', but in case
it's just "service@example.org", wrap it in angle brackets properly.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
When we're retrieving linked messages, make sure we don't clash with the
variable holding the overall mbox.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
This version returns a failure early when body modification is
recognized. This is especially useful if we have to shell out to gnupg
for validation.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|