211 lines
6.9 KiB
Plaintext
211 lines
6.9 KiB
Plaintext
Git performance tests
|
|
=====================
|
|
|
|
This directory holds performance testing scripts for git tools. The
|
|
first part of this document describes the various ways in which you
|
|
can run them.
|
|
|
|
When fixing the tools or adding enhancements, you are strongly
|
|
encouraged to add tests in this directory to cover what you are
|
|
trying to fix or enhance. The later part of this short document
|
|
describes how your test scripts should be organized.
|
|
|
|
|
|
Running Tests
|
|
-------------
|
|
|
|
The easiest way to run tests is to say "make". This runs all
|
|
the tests on the current git repository.
|
|
|
|
=== Running 2 tests in this tree ===
|
|
[...]
|
|
Test this tree
|
|
---------------------------------------------------------
|
|
0001.1: rev-list --all 0.54(0.51+0.02)
|
|
0001.2: rev-list --all --objects 6.14(5.99+0.11)
|
|
7810.1: grep worktree, cheap regex 0.16(0.16+0.35)
|
|
7810.2: grep worktree, expensive regex 7.90(29.75+0.37)
|
|
7810.3: grep --cached, cheap regex 3.07(3.02+0.25)
|
|
7810.4: grep --cached, expensive regex 9.39(30.57+0.24)
|
|
|
|
Output format is in seconds "Elapsed(User + System)"
|
|
|
|
You can compare multiple repositories and even git revisions with the
|
|
'run' script:
|
|
|
|
$ ./run . origin/next /path/to/git-tree p0001-rev-list.sh
|
|
|
|
where . stands for the current git tree. The full invocation is
|
|
|
|
./run [<revision|directory>...] [--] [<test-script>...]
|
|
|
|
A '.' argument is implied if you do not pass any other
|
|
revisions/directories.
|
|
|
|
You can also manually test this or another git build tree, and then
|
|
call the aggregation script to summarize the results:
|
|
|
|
$ ./p0001-rev-list.sh
|
|
[...]
|
|
$ ./run /path/to/other/git -- ./p0001-rev-list.sh
|
|
[...]
|
|
$ ./aggregate.perl . /path/to/other/git ./p0001-rev-list.sh
|
|
|
|
aggregate.perl has the same invocation as 'run', it just does not run
|
|
anything beforehand.
|
|
|
|
You can set the following variables (also in your config.mak):
|
|
|
|
GIT_PERF_REPEAT_COUNT
|
|
Number of times a test should be repeated for best-of-N
|
|
measurements. Defaults to 3.
|
|
|
|
GIT_PERF_MAKE_OPTS
|
|
Options to use when automatically building a git tree for
|
|
performance testing. E.g., -j6 would be useful. Passed
|
|
directly to make as "make $GIT_PERF_MAKE_OPTS".
|
|
|
|
GIT_PERF_MAKE_COMMAND
|
|
An arbitrary command that'll be run in place of the make
|
|
command, if set the GIT_PERF_MAKE_OPTS variable is
|
|
ignored. Useful in cases where source tree changes might
|
|
require issuing a different make command to different
|
|
revisions.
|
|
|
|
This can be (ab)used to monkeypatch or otherwise change the
|
|
tree about to be built. Note that the build directory can be
|
|
re-used for subsequent runs so the make command might get
|
|
executed multiple times on the same tree, but don't count on
|
|
any of that, that's an implementation detail that might change
|
|
in the future.
|
|
|
|
GIT_PERF_REPO
|
|
GIT_PERF_LARGE_REPO
|
|
Repositories to copy for the performance tests. The normal
|
|
repo should be at least git.git size. The large repo should
|
|
probably be about linux.git size for optimal results.
|
|
Both default to the git.git you are running from.
|
|
|
|
GIT_PERF_EXTRA
|
|
Boolean to enable additional tests. Most test scripts are
|
|
written to detect regressions between two versions of Git, and
|
|
the output will compare timings for individual tests between
|
|
those versions. Some scripts have additional tests which are not
|
|
run by default, that show patterns within a single version of
|
|
Git (e.g., performance of index-pack as the number of threads
|
|
changes). These can be enabled with GIT_PERF_EXTRA.
|
|
|
|
GIT_PERF_USE_SCALAR
|
|
Boolean indicating whether to register test repo(s) with Scalar
|
|
before executing tests.
|
|
|
|
You can also pass the options taken by ordinary git tests; the most
|
|
useful one is:
|
|
|
|
--root=<directory>::
|
|
Create "trash" directories used to store all temporary data during
|
|
testing under <directory>, instead of the t/ directory.
|
|
Using this option with a RAM-based filesystem (such as tmpfs)
|
|
can massively speed up the test suite.
|
|
|
|
|
|
Naming Tests
|
|
------------
|
|
|
|
The performance test files are named as:
|
|
|
|
pNNNN-commandname-details.sh
|
|
|
|
where N is a decimal digit. The same conventions for choosing NNNN as
|
|
for normal tests apply.
|
|
|
|
|
|
Writing Tests
|
|
-------------
|
|
|
|
The perf script starts much like a normal test script, except it
|
|
sources perf-lib.sh:
|
|
|
|
#!/bin/sh
|
|
#
|
|
# Copyright (c) 2005 Junio C Hamano
|
|
#
|
|
|
|
test_description='xxx performance test'
|
|
. ./perf-lib.sh
|
|
|
|
After that you will want to use some of the following:
|
|
|
|
test_perf_fresh_repo # sets up an empty repository
|
|
test_perf_default_repo # sets up a "normal" repository
|
|
test_perf_large_repo # sets up a "large" repository
|
|
|
|
test_perf_default_repo sub # ditto, in a subdir "sub"
|
|
|
|
test_checkout_worktree # if you need the worktree too
|
|
|
|
At least one of the first two is required!
|
|
|
|
You can use test_expect_success as usual. In both test_expect_success
|
|
and in test_perf, running "git" points to the version that is being
|
|
perf-tested. The $MODERN_GIT variable points to the git wrapper for the
|
|
currently checked-out version (i.e., the one that matches the t/perf
|
|
scripts you are running). This is useful if your setup uses commands
|
|
that only work with newer versions of git than what you might want to
|
|
test (but obviously your new commands must still create a state that can
|
|
be used by the older version of git you are testing).
|
|
|
|
For actual performance tests, use
|
|
|
|
test_perf 'descriptive string' '
|
|
command1 &&
|
|
command2
|
|
'
|
|
|
|
test_perf spawns a subshell, for lack of better options. This means
|
|
that
|
|
|
|
* you _must_ export all variables that you need in the subshell
|
|
|
|
* you _must_ flag all variables that you want to persist from the
|
|
subshell with 'test_export':
|
|
|
|
test_perf 'descriptive string' '
|
|
foo=$(git rev-parse HEAD) &&
|
|
test_export foo
|
|
'
|
|
|
|
The so-exported variables are automatically marked for export in the
|
|
shell executing the perf test. For your convenience, test_export is
|
|
the same as export in the main shell.
|
|
|
|
This feature relies on a bit of magic using 'set' and 'source'.
|
|
While we have tried to make sure that it can cope with embedded
|
|
whitespace and other special characters, it will not work with
|
|
multi-line data.
|
|
|
|
Rather than tracking the performance by run-time as `test_perf` does, you
|
|
may also track output size by using `test_size`. The stdout of the
|
|
function should be a single numeric value, which will be captured and
|
|
shown in the aggregated output. For example:
|
|
|
|
test_perf 'time foo' '
|
|
./foo >foo.out
|
|
'
|
|
|
|
test_size 'output size'
|
|
wc -c <foo.out
|
|
'
|
|
|
|
might produce output like:
|
|
|
|
Test origin HEAD
|
|
-------------------------------------------------------------
|
|
1234.1 time foo 0.37(0.79+0.02) 0.26(0.51+0.02) -29.7%
|
|
1234.2 output size 4.3M 3.6M -14.7%
|
|
|
|
The item being measured (and its units) is up to the test; the context
|
|
and the test title should make it clear to the user whether bigger or
|
|
smaller numbers are better. Unlike test_perf, the test code will only be
|
|
run once, since output sizes tend to be more deterministic than timings.
|