Commit Graph

21464 Commits (v2.43.0-rc1)

Author SHA1 Message Date
Junio C Hamano 8ed4eb7538 Merge branch 'tb/rev-list-unpacked-fix'
"git rev-list --unpacked --objects" failed to exclude packed
non-commit objects, which has been corrected.

* tb/rev-list-unpacked-fix:
  pack-bitmap: drop --unpacked non-commit objects from results
  list-objects: drop --unpacked non-commit objects from results
2023-11-08 15:04:42 +09:00
Junio C Hamano c732f7430d Merge branch 'ps/leakfixes'
Leakfix.

* ps/leakfixes:
  setup: fix leaking repository format
  setup: refactor `upgrade_repository_format()` to have common exit
  shallow: fix memory leak when registering shallow roots
  test-bloom: stop setting up Git directory twice
2023-11-08 15:04:41 +09:00
Junio C Hamano a8e2394704 Merge branch 'jc/test-i18ngrep'
Another step to deprecate test_i18ngrep.

* jc/test-i18ngrep:
  tests: teach callers of test_i18ngrep to use test_grep
  test framework: further deprecate test_i18ngrep
2023-11-08 11:04:02 +09:00
Junio C Hamano 259e30d2bb Merge branch 'bc/merge-file-object-input'
"git merge-file" learns a mode to read three contents to be merged
from blob objects.

* bc/merge-file-object-input:
  merge-file: add an option to process object IDs
  git-merge-file doc: drop "-file" from argument placeholders
2023-11-08 11:04:01 +09:00
Junio C Hamano 57e216d03d Merge branch 'kn/rev-list-missing-fix'
"git rev-list --missing" did not work for missing commit objects,
which has been corrected.

* kn/rev-list-missing-fix:
  rev-list: add commit object support in `--missing` option
  rev-list: move `show_commit()` to the bottom
  revision: rename bit to `do_not_die_on_missing_objects`
2023-11-08 11:04:01 +09:00
Junio C Hamano d8972a5abd Merge branch 'ps/show-ref'
Teach "git show-ref" a mode to check the existence of a ref.

* ps/show-ref:
  t: use git-show-ref(1) to check for ref existence
  builtin/show-ref: add new mode to check for reference existence
  builtin/show-ref: explicitly spell out different modes in synopsis
  builtin/show-ref: ensure mutual exclusiveness of subcommands
  builtin/show-ref: refactor options for patterns subcommand
  builtin/show-ref: stop using global vars for `show_one()`
  builtin/show-ref: stop using global variable to count matches
  builtin/show-ref: refactor `--exclude-existing` options
  builtin/show-ref: fix dead code when passing patterns
  builtin/show-ref: fix leaking string buffer
  builtin/show-ref: split up different subcommands
  builtin/show-ref: convert pattern to a local variable
2023-11-08 11:04:00 +09:00
Junio C Hamano 42b87f7ee6 Merge branch 'ps/do-not-trust-commit-graph-blindly-for-existence'
The codepath to traverse the commit-graph learned to notice that a
commit is missing (e.g., corrupt repository lost an object), even
though it knows something about the commit (like its parents) from
what is in commit-graph.

* ps/do-not-trust-commit-graph-blindly-for-existence:
  commit: detect commits that exist in commit-graph but not in the ODB
  commit-graph: introduce envvar to disable commit existence checks
2023-11-08 11:03:59 +09:00
Junio C Hamano 650963cfd0 Merge branch 'jk/chunk-bounds'
Test portability fix.

* jk/chunk-bounds:
  t: avoid perl's pack/unpack "Q" specifier
2023-11-08 11:03:58 +09:00
Taylor Blau 7b3c8e9f38 pack-bitmap: drop --unpacked non-commit objects from results
When performing revision queries with `--objects` and
`--use-bitmap-index`, the output may incorrectly contain objects which
are packed, even when the `--unpacked` option is given. This affects
traversals, but also other querying operations, like `--count`,
`--disk-usage`, etc.

Like in the previous commit, the fix is to exclude those objects from
the result set before they are shown to the user (or, in this case,
before the bitmap containing the result of the traversal is enumerated
and its objects listed).

This is performed by a new function in pack-bitmap.c, called
`filter_packed_objects_from_bitmap()`. Note that we do not have to
inspect individual bits in the result bitmap, since we know that the
first N (where N is the number of objects in the bitmap's pack/MIDX)
bits correspond to objects which packed by definition.

In other words, for an object to have a bitmap position (not in the
extended index), it must appear in either the bitmap's pack or one of
the packs in its MIDX.

This presents an appealing optimization to us, which is that we can
simply memset() the corresponding number of `eword_t`'s to zero,
provided that we handle any objects which spill into the next word (but
don't occupy all 64 bits of the word itself).

We only have to handle objects in the bitmap's extended index. These
objects may (or may not) appear in one or more pack(s). Since these
objects are known to not appear in either the bitmap's MIDX or pack,
they may be stored as loose, appear in other pack(s), or both.

Before returning a bitmap containing the result of the traversal back to
the caller, drop any bits from the extended index which appear in one or
more packs. This implements the correct behavior for rev-list operations
which use the bitmap index to compute their result.

Co-authored-by: Jeff King <peff@peff.net>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-11-07 11:23:52 +09:00
Taylor Blau 4263f9279e list-objects: drop --unpacked non-commit objects from results
In git-rev-list(1), we describe the `--unpacked` option as:

    Only useful with `--objects`; print the object IDs that are not in
    packs.

This is true of commits, which we discard via get_commit_action(), but
not of the objects they reach. So if we ask for an --objects traversal
with --unpacked, we may get arbitrarily many objects which are indeed
packed.

I am nearly certain this behavior dates back to the introduction of
`--unpacked` via 12d2a18780 ("git rev-list --unpacked" shows only
unpacked commits, 2005-07-03), but I couldn't get that revision of Git
to compile for me. At least as early as v2.0.0 this has been subtly
broken:

    $ git.compile --version
    git version 2.0.0

    $ git.compile rev-list --objects --all --unpacked
    72791fe96c93f9ec5c311b8bc966ab349b3b5bbe
    05713d991c18bbeef7e154f99660005311b5004d v1.0
    153ed8b7719c6f5a68ce7ffc43133e95a6ac0fdb
    8e4020bb5a8d8c873b25de15933e75cc0fc275df one
    9200b628cf9dc883a85a7abc8d6e6730baee589c two
    3e6b46e1b7e3b91acce99f6a823104c28aae0b58 unpacked.t

There, only the first, third, and sixth entries are loose, with the
remaining set of objects belonging to at least one pack.

The implications for this are relatively benign: bare 'git repack'
invocations which invoke pack-objects with --unpacked are impacted, and
at worst we'll store a few extra objects that should have been excluded.

Arguably changing this behavior is a backwards-incompatible change,
since it alters the set of objects emitted from rev-list queries with
`--objects` and `--unpacked`. But I argue that this change is still
sensible, since the existing implementation deviates from
clearly-written documentation.

The fix here is straightforward: avoid showing any non-commit objects
which are contained in packs by discarding them within list-objects.c,
before they are shown to the user. Note that similar treatment for
`list-objects.c::show_commit()` is not needed, since that case is
already handled by `revision.c::get_commit_action()`.

Co-authored-by: Jeff King <peff@peff.net>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-11-07 11:23:51 +09:00
Junio C Hamano a362332c56 Merge branch 'rc/trace-upload-pack'
Trace2 update.

* rc/trace-upload-pack:
  upload-pack: add tracing for fetches
2023-11-07 10:26:45 +09:00
Junio C Hamano 840bd1c9ef Merge branch 'es/bugreport-no-extra-arg'
"git bugreport" learned to complain when it received a command line
argument that it will not use.

* es/bugreport-no-extra-arg:
  bugreport: reject positional arguments
  t0091-bugreport: stop using i18ngrep
2023-11-07 10:26:44 +09:00
Junio C Hamano 00f372e2a4 Merge branch 'ms/send-email-validate-fix'
"git send-email" did not have certain pieces of data computed yet
when it tried to validate the outging messages and its recipient
addresses, which has been sorted out.

* ms/send-email-validate-fix:
  send-email: move validation code below process_address_list
2023-11-07 10:26:44 +09:00
Junio C Hamano dbffe54f8a Merge branch 'rs/reflog-expire-single-worktree-fix'
"git reflog expire --single-worktree" has been broken for the past
20 months or so, which has been corrected.

* rs/reflog-expire-single-worktree-fix:
  reflog: fix expire --single-worktree
2023-11-07 10:26:44 +09:00
Junio C Hamano 5f11becce0 Merge branch 'rs/parse-options-cmdmode'
parse-options improvements for OPT_CMDMODE options.

* rs/parse-options-cmdmode:
  am: simplify --show-current-patch handling
  parse-options: make CMDMODE errors more precise
2023-11-07 10:26:43 +09:00
Junio C Hamano 2d2cd0a1bc Merge branch 'jc/grep-f-relative-to-cwd'
"cd sub && git grep -f patterns" tried to read "patterns" file at
the top level of the working tree; it has been corrected to read
"sub/patterns" instead.

* jc/grep-f-relative-to-cwd:
  grep: -f <path> is relative to $cwd
2023-11-07 10:26:43 +09:00
Patrick Steinhardt 568cc818cc shallow: fix memory leak when registering shallow roots
When registering shallow roots, we unset the list of parents of the
to-be-registered commit if it's already been parsed. This causes us to
leak memory though because we never free this list. Fix this.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-11-07 08:51:41 +09:00
Patrick Steinhardt 40e9136ff6 test-bloom: stop setting up Git directory twice
We're setting up the Git directory twice in the `test-tool bloom`
helper, once at the beginning of `cmd_bloom()` and once in the local
subcommand implementation `get_bloom_filter_for_commit()`. This can lead
to memory leaks as we'll overwrite variables of `the_repository` with
newly allocated data structures. On top of that it's simply unnecessary.

Fix this by only setting up the Git directory once.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-11-07 08:51:40 +09:00
Jeff King 4815c3c4b2 t: avoid perl's pack/unpack "Q" specifier
The perl script introduced by 86b008ee61 (t: add library for munging
chunk-format files, 2023-10-09) uses pack("Q") and unpack("Q") to read
and write 64-bit values ("quadwords" in perl parlance) from the on-disk
chunk files. However, some builds of perl may not support 64-bit
integers at all, and throw an exception here. While some 32-bit
platforms may still support 64-bit integers in perl (such as our linux32
CI environment), others reportedly don't (the NonStop 32-bit builds).

We can work around this by treating the 64-bit values as two 32-bit
values. We can't ever combine them into a single 64-bit value, but in
practice this is OK. These are representing file offsets, and our files
are much smaller than 4GB. So the upper half of the 64-bit value will
always be 0.

We can just introduce a few helper functions which perform the
translation and double-check our assumptions.

Reported-by: Randall S. Becker <randall.becker@nexbridge.ca>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-11-04 15:54:25 +09:00
Junio C Hamano 6789275d37 tests: teach callers of test_i18ngrep to use test_grep
They are equivalents and the former still exists, so as long as the
only change this commit makes are to rewrite test_i18ngrep to
test_grep, there won't be any new bug, even if there still are
callers of test_i18ngrep remaining in the tree, or when merged to
other topics that add new uses of test_i18ngrep.

This patch was produced more or less with

    git grep -l -e 'test_i18ngrep ' 't/t[0-9][0-9][0-9][0-9]-*.sh' |
    xargs perl -p -i -e 's/test_i18ngrep /test_grep /'

and a good way to sanity check the result yourself is to run the
above in a checkout of c4603c1c (test framework: further deprecate
test_i18ngrep, 2023-10-31) and compare the resulting working tree
contents with the result of applying this patch to the same commit.
You'll see that test_i18ngrep in a few t/lib-*.sh files corrected,
in addition to the manual reproduction.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-11-02 17:13:44 +09:00
Junio C Hamano f92cea12c9 Merge branch 'jk/decoration-and-other-leak-fixes' into maint-2.42
Leakfix.

* jk/decoration-and-other-leak-fixes:
  daemon: free listen_addr before returning
  revision: clear decoration structs during release_revisions()
  decorate: add clear_decoration() function
2023-11-02 16:53:26 +09:00
Junio C Hamano 71c614b9a2 Merge branch 'ob/t3404-typofix' into maint-2.42
Code clean-up.

* ob/t3404-typofix:
  t3404-rebase-interactive.sh: fix typos in title of a rewording test
2023-11-02 16:53:24 +09:00
Junio C Hamano 535b30eb58 Merge branch 'bc/more-git-var' into maint-2.42
Fix-up for a topic that already has graduated.

* bc/more-git-var:
  var: avoid a segmentation fault when `HOME` is unset
2023-11-02 16:53:23 +09:00
Junio C Hamano 43af21409e Merge branch 'ch/t6300-verify-commit-test-cleanup' into maint-2.42
Test clean-up.

* ch/t6300-verify-commit-test-cleanup:
  t/t6300: drop magic filtering
  t/lib-gpg: forcibly run a trustdb update
2023-11-02 16:53:22 +09:00
Junio C Hamano 8db7d2d6bd Merge branch 'ob/t9001-indent-fix' into maint-2.42
Test style fix.

* ob/t9001-indent-fix:
  t9001: fix indentation in test_no_confirm()
2023-11-02 16:53:21 +09:00
Junio C Hamano 07011e1480 Merge branch 'jk/test-pass-ubsan-options-to-http-test' into maint-2.42
UBSAN options were not propagated through the test framework to git
run via the httpd, unlike ASAN options, which has been corrected.

* jk/test-pass-ubsan-options-to-http-test:
  test-lib: set UBSAN_OPTIONS to match ASan
2023-11-02 16:53:20 +09:00
Junio C Hamano 2fd4378d64 Merge branch 'js/diff-cached-fsmonitor-fix' into maint-2.42
"git diff --cached" codepath did not fill the necessary stat
information for a file when fsmonitor knows it is clean and ended
up behaving as if it is not clean, which has been corrected.

* js/diff-cached-fsmonitor-fix:
  diff-lib: fix check_removed when fsmonitor is on
2023-11-02 16:53:19 +09:00
Junio C Hamano bfb8376d68 Merge branch 'pw/diff-no-index-from-named-pipes' into maint-2.42
"git diff --no-index -R <(one) <(two)" did not work correctly,
which has been corrected.

* pw/diff-no-index-from-named-pipes:
  diff --no-index: fix -R with stdin
2023-11-02 16:53:18 +09:00
Junio C Hamano d12df942ba Merge branch 'js/complete-checkout-t' into maint-2.42
The completion script (in contrib/) has been taught to treat the
"-t" option to "git checkout" and "git switch" just like the
"--track" option, to complete remote-tracking branches.

* js/complete-checkout-t:
  completion(switch/checkout): treat --track and -t the same
2023-11-02 16:53:18 +09:00
Junio C Hamano a70f725c06 Merge branch 'pw/rebase-i-after-failure' into maint-2.42
Various fixes to the behaviour of "rebase -i" when the command got
interrupted by conflicting changes.
cf. <6b927687-cf6e-d73e-78fb-bd4f46736928@gmx.de>

* pw/rebase-i-after-failure:
  rebase -i: fix adding failed command to the todo list
  rebase --continue: refuse to commit after failed command
  rebase: fix rewritten list for failed pick
  sequencer: factor out part of pick_commits()
  sequencer: use rebase_path_message()
  rebase -i: remove patch file after conflict resolution
  rebase -i: move unlink() calls
2023-11-02 16:53:17 +09:00
Junio C Hamano d97034b0a1 Merge branch 'ks/ref-filter-sort-numerically' into maint-2.42
"git for-each-ref --sort='contents:size'" sorts the refs according
to size numerically, giving a ref that points at a blob twelve-byte
(12) long before showing a blob hundred-byte (100) long.

* ks/ref-filter-sort-numerically:
  ref-filter: sort numerically when ":size" is used
2023-11-02 16:53:17 +09:00
Junio C Hamano 57b52cec46 Merge branch 'jk/diff-result-code-cleanup' into maint-2.42
"git diff --no-such-option" and other corner cases around the exit
status of the "diff" command has been corrected.

* jk/diff-result-code-cleanup:
  diff: drop useless "status" parameter from diff_result_code()
  diff: drop useless return values in git-diff helpers
  diff: drop useless return from run_diff_{files,index} functions
  diff: die when failing to read index in git-diff builtin
  diff: show usage for unknown builtin_diff_files() options
  diff-files: avoid negative exit value
  diff: spell DIFF_INDEX_CACHED out when calling run_diff_index()
2023-11-02 16:53:16 +09:00
Junio C Hamano 7f8314f277 Merge branch 'ts/unpacklimit-config-fix' into maint-2.42
transfer.unpackLimit ought to be used as a fallback, but overrode
fetch.unpackLimit and receive.unpackLimit instead.

* ts/unpacklimit-config-fix:
  transfer.unpackLimit: fetch/receive.unpackLimit takes precedence
2023-11-02 16:53:16 +09:00
Junio C Hamano 8764491463 Merge branch 'jc/diff-exit-code-with-w-fixes' into maint-2.42
"git diff -w --exit-code" with various options did not work
correctly, which is being addressed.

* jc/diff-exit-code-with-w-fixes:
  diff: the -w option breaks --exit-code for --raw and other output modes
  t4040: remove test that succeeded for a wrong reason
  diff: teach "--stat -w --exit-code" to notice differences
  diff: mode-only change should be noticed by "--patch -w --exit-code"
  diff: move the fallback "--exit-code" code down
2023-11-02 16:53:15 +09:00
Junio C Hamano 1ea39ad467 Merge branch 'tb/commit-graph-verify-fix' into maint-2.42
The commit-graph verification code that detects mixture of zero and
non-zero generation numbers has been updated.

* tb/commit-graph-verify-fix:
  commit-graph: avoid repeated mixed generation number warnings
  t/t5318-commit-graph.sh: test generation zero transitions during fsck
  commit-graph: verify swapped zero/non-zero generation cases
  commit-graph: introduce `commit_graph_generation_from_graph()`
2023-11-02 16:53:15 +09:00
Junio C Hamano 50758312f2 Merge branch 'ds/scalar-updates' into maint-2.42
Scalar updates.

* ds/scalar-updates:
  scalar reconfigure: help users remove buggy repos
  setup: add discover_git_directory_reason()
  scalar: add --[no-]src option
2023-11-02 16:53:15 +09:00
Junio C Hamano 396a167bd4 Merge branch 'mp/rebase-label-length-limit' into maint-2.42
Overly long label names used in the sequencer machinery are now
chopped to fit under filesystem limitation.

* mp/rebase-label-length-limit:
  rebase: allow overriding the maximal length of the generated labels
  sequencer: truncate labels to accommodate loose refs
2023-11-02 16:53:14 +09:00
Junio C Hamano 6d68ab0819 Merge branch 'jk/test-lsan-denoise-output' into maint-2.42
Tests with LSan from time to time seem to emit harmless message
that makes our tests unnecessarily flakey; we work it around by
filtering the uninteresting output.

* jk/test-lsan-denoise-output:
  test-lib: ignore uninteresting LSan output
2023-11-02 16:53:14 +09:00
brian m. carlson e1068f0ad4 merge-file: add an option to process object IDs
git merge-file knows how to merge files on the file system already.  It
would be helpful, however, to allow it to also merge single blobs.
Teach it an `--object-id` option which means that its arguments are
object IDs and not files to allow it to do so.

We handle the empty blob specially since read_mmblob doesn't read it
directly and otherwise users cannot specify an empty ancestor.

Signed-off-by: brian m. carlson <bk2204@github.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-11-02 08:51:40 +09:00
Patrick Steinhardt 0497e6c611 t: use git-show-ref(1) to check for ref existence
Convert tests that use `test_path_is_file` and `test_path_is_missing` to
instead use a set of helpers `test_ref_exists` and `test_ref_missing`.
These helpers are implemented via the newly introduced `git show-ref
--exists` command. Thus, we can avoid intimate knowledge of how the ref
backend stores references on disk.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-11-01 12:09:01 +09:00
Patrick Steinhardt 9080a7f178 builtin/show-ref: add new mode to check for reference existence
While we have multiple ways to show the value of a given reference, we
do not have any way to check whether a reference exists at all. While
commands like git-rev-parse(1) or git-show-ref(1) can be used to check
for reference existence in case the reference resolves to something
sane, neither of them can be used to check for existence in some other
scenarios where the reference does not resolve cleanly:

    - References which have an invalid name cannot be resolved.

    - References to nonexistent objects cannot be resolved.

    - Dangling symrefs can be resolved via git-symbolic-ref(1), but this
      requires the caller to special case existence checks depending on
      whether or not a reference is symbolic or direct.

Furthermore, git-rev-list(1) and other commands do not let the caller
distinguish easily between an actually missing reference and a generic
error.

Taken together, this seems like sufficient motivation to introduce a
separate plumbing command to explicitly check for the existence of a
reference without trying to resolve its contents.

This new command comes in the form of `git show-ref --exists`. This
new mode will exit successfully when the reference exists, with a
specific exit code of 2 when it does not exist, or with 1 when there
has been a generic error.

Note that the only way to properly implement this command is by using
the internal `refs_read_raw_ref()` function. While the public function
`refs_resolve_ref_unsafe()` can be made to behave in the same way by
passing various flags, it does not provide any way to obtain the errno
with which the reference backend failed when reading the reference. As
such, it becomes impossible for us to distinguish generic errors from
the explicit case where the reference wasn't found.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-11-01 12:09:01 +09:00
Patrick Steinhardt 199970e72f builtin/show-ref: ensure mutual exclusiveness of subcommands
The git-show-ref(1) command has three different modes, of which one is
implicit and the other two can be chosen explicitly by passing a flag.
But while these modes are standalone and cause us to execute completely
separate code paths, we gladly accept the case where a user asks for
both `--exclude-existing` and `--verify` at the same time even though it
is not obvious what will happen. Spoiler: we ignore `--verify` and
execute the `--exclude-existing` mode.

Let's explicitly detect this invalid usage and die in case both modes
were requested.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-11-01 12:09:00 +09:00
Karthik Nayak 9830926c7d rev-list: add commit object support in `--missing` option
The `--missing` object option in rev-list currently works only with
missing blobs/trees. For missing commits the revision walker fails with
a fatal error.

Let's extend the functionality of `--missing` option to also support
commit objects. This is done by adding a `missing_objects` field to
`rev_info`. This field is an `oidset` to which we'll add the missing
commits as we encounter them. The revision walker will now continue the
traversal and call `show_commit()` even for missing commits. In rev-list
we can then check if the commit is a missing commit and call the
existing code for parsing `--missing` objects.

A scenario where this option would be used is to find the boundary
objects between different object directories. Consider a repository with
a main object directory (GIT_OBJECT_DIRECTORY) and one or more alternate
object directories (GIT_ALTERNATE_OBJECT_DIRECTORIES). In such a
repository, using the `--missing=print` option while disabling the
alternate object directory allows us to find the boundary objects
between the main and alternate object directory.

Helped-by: Patrick Steinhardt <ps@pks.im>
Helped-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-11-01 12:07:18 +09:00
Junio C Hamano 922cc26e41 Merge branch 'ps/do-not-trust-commit-graph-blindly-for-existence' into kn/rev-list-missing-fix
* ps/do-not-trust-commit-graph-blindly-for-existence:
  commit: detect commits that exist in commit-graph but not in the ODB
  commit-graph: introduce envvar to disable commit existence checks
2023-11-01 12:06:55 +09:00
Patrick Steinhardt 7a5d604443 commit: detect commits that exist in commit-graph but not in the ODB
Commit graphs can become stale and contain references to commits that do
not exist in the object database anymore. Theoretically, this can lead
to a scenario where we are able to successfully look up any such commit
via the commit graph even though such a lookup would fail if done via
the object database directly.

As the commit graph is mostly intended as a sort of cache to speed up
parsing of commits we do not want to have diverging behaviour in a
repository with and a repository without commit graphs, no matter
whether they are stale or not. As commits are otherwise immutable, the
only thing that we really need to care about is thus the presence or
absence of a commit.

To address potentially stale commit data that may exist in the graph,
our `lookup_commit_in_graph()` function will check for the commit's
existence in both the commit graph, but also in the object database. So
even if we were able to look up the commit's data in the graph, we would
still pretend as if the commit didn't exist if it is missing in the
object database.

We don't have the same safety net in `parse_commit_in_graph_one()`
though. This function is mostly used internally in "commit-graph.c"
itself to validate the commit graph, and this usage is fine. We do
expose its functionality via `parse_commit_in_graph()` though, which
gets called by `repo_parse_commit_internal()`, and that function is in
turn used in many places in our codebase.

For all I can see this function is never used to directly turn an object
ID into a commit object without additional safety checks before or after
this lookup. What it is being used for though is to walk history via the
parent chain of commits. So when commits in the parent chain of a graph
walk are missing it is possible that we wouldn't notice if that missing
commit was part of the commit graph. Thus, a query like `git rev-parse
HEAD~2` can succeed even if the intermittent commit is missing.

It's unclear whether there are additional ways in which such stale
commit graphs can lead to problems. In any case, it feels like this is a
bigger bug waiting to happen when we gain additional direct or indirect
callers of `repo_parse_commit_internal()`. So let's fix the inconsistent
behaviour by checking for object existence via the object database, as
well.

This check of course comes with a performance penalty. The following
benchmarks have been executed in a clone of linux.git with stable tags
added:

    Benchmark 1: git -c core.commitGraph=true rev-list --topo-order --all (git = master)
      Time (mean ± σ):      2.913 s ±  0.018 s    [User: 2.363 s, System: 0.548 s]
      Range (min … max):    2.894 s …  2.950 s    10 runs

    Benchmark 2: git -c core.commitGraph=true rev-list --topo-order --all (git = pks-commit-graph-inconsistency)
      Time (mean ± σ):      3.834 s ±  0.052 s    [User: 3.276 s, System: 0.556 s]
      Range (min … max):    3.780 s …  3.961 s    10 runs

    Benchmark 3: git -c core.commitGraph=false rev-list --topo-order --all (git = master)
      Time (mean ± σ):     13.841 s ±  0.084 s    [User: 13.152 s, System: 0.687 s]
      Range (min … max):   13.714 s … 13.995 s    10 runs

    Benchmark 4: git -c core.commitGraph=false rev-list --topo-order --all (git = pks-commit-graph-inconsistency)
      Time (mean ± σ):     13.762 s ±  0.116 s    [User: 13.094 s, System: 0.667 s]
      Range (min … max):   13.645 s … 14.038 s    10 runs

    Summary
      git -c core.commitGraph=true rev-list --topo-order --all (git = master) ran
        1.32 ± 0.02 times faster than git -c core.commitGraph=true rev-list --topo-order --all (git = pks-commit-graph-inconsistency)
        4.72 ± 0.05 times faster than git -c core.commitGraph=false rev-list --topo-order --all (git = pks-commit-graph-inconsistency)
        4.75 ± 0.04 times faster than git -c core.commitGraph=false rev-list --topo-order --all (git = master)

We look at a ~30% regression in general, but in general we're still a
whole lot faster than without the commit graph. To counteract this, the
new check can be turned off with the `GIT_COMMIT_GRAPH_PARANOIA` envvar.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-11-01 12:04:06 +09:00
Patrick Steinhardt e04838ea82 commit-graph: introduce envvar to disable commit existence checks
Our `lookup_commit_in_graph()` helper tries to look up commits from the
commit graph and, if it doesn't exist there, falls back to parsing it
from the object database instead. This is intended to speed up the
lookup of any such commit that exists in the database. There is an edge
case though where the commit exists in the graph, but not in the object
database. To avoid returning such stale commits the helper function thus
double checks that any such commit parsed from the graph also exists in
the object database. This makes the function safe to use even when
commit graphs aren't updated regularly.

We're about to introduce the same pattern into other parts of our code
base though, namely `repo_parse_commit_internal()`. Here the extra
sanity check is a bit of a tougher sell: `lookup_commit_in_graph()` was
a newly introduced helper, and as such there was no performance hit by
adding this sanity check. If we added `repo_parse_commit_internal()`
with that sanity check right from the beginning as well, this would
probably never have been an issue to begin with. But by retrofitting it
with this sanity check now we do add a performance regression to
preexisting code, and thus there is a desire to avoid this or at least
give an escape hatch.

In practice, there is no inherent reason why either of those functions
should have the sanity check whereas the other one does not: either both
of them are able to detect this issue or none of them should be. This
also means that the default of whether we do the check should likely be
the same for both. To err on the side of caution, we thus rather want to
make `repo_parse_commit_internal()` stricter than to loosen the checks
that we already have in `lookup_commit_in_graph()`.

The escape hatch is added in the form of a new GIT_COMMIT_GRAPH_PARANOIA
environment variable that mirrors GIT_REF_PARANOIA. If enabled, which is
the default, we will double check that commits looked up in the commit
graph via `lookup_commit_in_graph()` also exist in the object database.
This same check will also be added in `repo_parse_commit_internal()`.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-11-01 12:04:06 +09:00
Junio C Hamano 2e87fca189 test framework: further deprecate test_i18ngrep
As an attempt to come up with a useful mechanism to ensure that
certain messages are left untranslated [*], we earlier wrote
GIT_TEST_GETTEXT_POISON off as a failed experiment.

But the output from the test helper was easier to use while
debugging failed tests, compared to the same test writtein with the
plain-vanilla "grep".  Especially when a test that expects a certain
string to appear in the output (e.g. "this test must fail with this
message") fails, "grep message output" would just silently fail and
in a &&-chained sequence of commands, it is hard to tell which step
failed.  test_i18ngrep explicitly said "we wanted to see a line that
match this pattern but did not see a hit in this file".

What we have as test_i18ngrep in our tree still retains this verbose
output (even though we got rid of the "poison" support).  Let's
rename it to test_grep (because it is no longer about i18n at all)
and then make test_i18ngrep a thin wrapper around it.  Existing
callers of test_i18ngrep can be mechanically rewritten to instead
use test_grep over time, but it does not have to be done in this
commit.

[Footnote]

 * The idea was that human-facing messages are often translated, but
   there are messages that should never be translated.  We use
   "grep" only for the latter kind of messages, and then run tests
   in "poison" mode that spew garbage for translatable messages.  If
   such a test run fails, it means these messages tested with "grep"
   were marked for translation by mistake.  test_i18ngrep was to be
   used for other messages that are to be translated, and was to
   always "succeed" when runing under the "poison" mode.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-10-31 14:24:35 +09:00
Junio C Hamano ece54894fe Merge branch 'ii/branch-error-messages-update'
Error message update.

* ii/branch-error-messages-update:
  builtin/branch.c: adjust error messages to coding guidelines
2023-10-31 12:57:44 +09:00
Robert Coup b8f58c200c upload-pack: add tracing for fetches
Information on how users are accessing hosted repositories can be
helpful to server operators. For example, being able to broadly
differentiate between fetches and initial clones; the use of shallow
repository features; or partial clone filters.

a29263c (fetch-pack: add tracing for negotiation rounds, 2022-08-02)
added some information on have counts to fetch-pack itself to help
diagnose negotiation; but from a git-upload-pack (server) perspective,
there's no means of accessing such information without using
GIT_TRACE_PACKET to examine the protocol packets.

Improve this by emitting a Trace2 JSON event from upload-pack with
summary information on the contents of a fetch request.

* haves, wants, and want-ref counts can help determine (broadly) between
  fetches and clones, and the use of single-branch, etc.
* shallow clone depth, tip counts, and deepening options.
* any partial clone filter type.

Signed-off-by: Robert Coup <robert@coup.net.nz>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-10-30 21:43:21 +09:00
Junio C Hamano 5006bfc1f5 Merge branch 'jk/send-email-fix-addresses-from-composed-messages'
The codepath to handle recipient addresses `git send-email
--compose` learns from the user was completely broken, which has
been corrected.

* jk/send-email-fix-addresses-from-composed-messages:
  send-email: handle to/cc/bcc from --compose message
  Revert "send-email: extract email-parsing code into a subroutine"
  doc/send-email: mention handling of "reply-to" with --compose
2023-10-30 07:09:59 +09:00