mirror
/
git
mirror of https://git.kernel.org/pub/scm/git/git.git
1
0
Fork 0
Code Releases Activity

Merge branch 'tb/incremental-midx-part-2' into next 

Incrementally updating multi-pack index files.

* tb/incremental-midx-part-2:
  midx: implement writing incremental MIDX bitmaps
  pack-bitmap.c: use `ewah_or_iterator` for type bitmap iterators
  pack-bitmap.c: keep track of each layer's type bitmaps
  ewah: implement `struct ewah_or_iterator`
  pack-bitmap.c: apply pseudo-merge commits with incremental MIDXs
  pack-bitmap.c: compute disk-usage with incremental MIDXs
  pack-bitmap.c: teach `rev-list --test-bitmap` about incremental MIDXs
  pack-bitmap.c: support bitmap pack-reuse with incremental MIDXs
  pack-bitmap.c: teach `show_objects_for_type()` about incremental MIDXs
  pack-bitmap.c: teach `bitmap_for_commit()` about incremental MIDXs
  pack-bitmap.c: open and store incremental bitmap layers
  pack-revindex: prepare for incremental MIDX bitmaps
  Documentation: describe incremental MIDX bitmaps
  Documentation: remove a "future work" item from the MIDX docs
next
Junio C Hamano 2025-03-29 16:43:42 +09:00
parent 4e9fbae111 27afc272c4
commit 1db17d1a9b
10 changed files with 589 additions and 132 deletions

82
Documentation/technical/multi-pack-index.adoc
View File

@ -164,19 +164,81 @@ objects_nr($H2) + objects_nr($H1) + i
(in the C implementation, this is often computed as `i +
m->num_objects_in_base`).
=== Pseudo-pack order for incremental MIDXs
The original implementation of multi-pack reachability bitmaps defined
the pseudo-pack order in linkgit:gitformat-pack[5] (see the section
titled "multi-pack-index reverse indexes") roughly as follows:
____
In short, a MIDX's pseudo-pack is the de-duplicated concatenation of
objects in packs stored by the MIDX, laid out in pack order, and the
packs arranged in MIDX order (with the preferred pack coming first).
____
In the incremental MIDX design, we extend this definition to include
objects from multiple layers of the MIDX chain. The pseudo-pack order
for incremental MIDXs is determined by concatenating the pseudo-pack
ordering for each layer of the MIDX chain in order. Formally two objects
`o1` and `o2` are compared as follows:
1. If `o1` appears in an earlier layer of the MIDX chain than `o2`, then
`o1` sorts ahead of `o2`.
2. Otherwise, if `o1` and `o2` appear in the same MIDX layer, and that
MIDX layer has no base, then if one of `pack(o1)` and `pack(o2)` is
preferred and the other is not, then the preferred one sorts ahead of
the non-preferred one. If there is a base layer (i.e. the MIDX layer
is not the first layer in the chain), then if `pack(o1)` appears
earlier in that MIDX layer's pack order, then `o1` sorts ahead of
`o2`. Likewise if `pack(o2)` appears earlier, then the opposite is
true.
3. Otherwise, `o1` and `o2` appear in the same pack, and thus in the
same MIDX layer. Sort `o1` and `o2` by their offset within their
containing packfile.
Note that the preferred pack is a property of the MIDX chain, not the
individual layers themselves. Fundamentally we could introduce a
per-layer preferred pack, but this is less relevant now that we can
perform multi-pack reuse across the set of packs in a MIDX.
=== Reachability bitmaps and incremental MIDXs
Each layer of an incremental MIDX chain may have its objects (and the
objects from any previous layer in the same MIDX chain) represented in
its own `*.bitmap` file.
The structure of a `*.bitmap` file belonging to an incremental MIDX
chain is identical to that of a non-incremental MIDX bitmap, or a
classic single-pack bitmap. Since objects are added to the end of the
incremental MIDX's pseudo-pack order (see above), it is possible to
extend a bitmap when appending to the end of a MIDX chain.
(Note: it is possible likewise to compress a contiguous sequence of MIDX
incremental layers, and their `*.bitmap` files into a single layer and
`*.bitmap`, but this is not yet implemented.)
The object positions used are global within the pseudo-pack order, so
subsequent layers will have, for example, `m->num_objects_in_base`
number of `0` bits in each of their four type bitmaps. This follows from
the fact that we only write type bitmap entries for objects present in
the layer immediately corresponding to the bitmap).
Note also that only the bitmap pertaining to the most recent layer in an
incremental MIDX chain is used to store reachability information about
the interesting and uninteresting objects in a reachability query.
Earlier bitmap layers are only used to look up commit and pseudo-merge
bitmaps from that layer, as well as the type-level bitmaps for objects
in that layer.
To simplify the implementation, type-level bitmaps are iterated
simultaneously, and their results are OR'd together to avoid recursively
calling internal bitmap functions.
Future Work
-----------
- The multi-pack-index allows many packfiles, especially in a context
where repacking is expensive (such as a very large repo), or
unexpected maintenance time is unacceptable (such as a high-demand
build machine). However, the multi-pack-index needs to be rewritten
in full every time. We can extend the format to be incremental, so
writes are fast. By storing a small "tip" multi-pack-index that
points to large "base" MIDX files, we can keep writes fast while
still reducing the number of binary searches required for object
lookups.
- If the multi-pack-index is extended to store a "stable object order"
(a function Order(hash) = integer that is constant for a given hash,
even as the multi-pack-index is updated) then MIDX bitmaps could be

3
builtin/pack-objects.c
View File

@ -1398,7 +1398,8 @@ static void write_pack_file(void)
if (write_bitmap_index) {
bitmap_writer_init(&bitmap_writer,
the_repository, &to_pack);
the_repository, &to_pack,
NULL);
bitmap_writer_set_checksum(&bitmap_writer, hash);
bitmap_writer_build_type_index(&bitmap_writer,
written_list);

33
ewah/ewah_bitmap.c
View File

@ -371,6 +371,39 @@ void ewah_iterator_init(struct ewah_iterator *it, struct ewah_bitmap *parent)
read_new_rlw(it);
}
void ewah_or_iterator_init(struct ewah_or_iterator *it,
struct ewah_bitmap **parents, size_t nr)
{
size_t i;
memset(it, 0, sizeof(*it));
ALLOC_ARRAY(it->its, nr);
for (i = 0; i < nr; i++)
ewah_iterator_init(&it->its[it->nr++], parents[i]);
}
int ewah_or_iterator_next(eword_t *next, struct ewah_or_iterator *it)
{
eword_t buf, out = 0;
size_t i;
int ret = 0;
for (i = 0; i < it->nr; i++)
if (ewah_iterator_next(&buf, &it->its[i])) {
out |= buf;
ret = 1;
}
*next = out;
return ret;
}
void ewah_or_iterator_release(struct ewah_or_iterator *it)
{
free(it->its);
}
void ewah_xor(
struct ewah_bitmap *ewah_i,
struct ewah_bitmap *ewah_j,

12
ewah/ewok.h
View File

@ -148,6 +148,18 @@ void ewah_iterator_init(struct ewah_iterator *it, struct ewah_bitmap *parent);
*/
int ewah_iterator_next(eword_t *next, struct ewah_iterator *it);
struct ewah_or_iterator {
struct ewah_iterator *its;
size_t nr;
};
void ewah_or_iterator_init(struct ewah_or_iterator *it,
struct ewah_bitmap **parents, size_t nr);
int ewah_or_iterator_next(eword_t *next, struct ewah_or_iterator *it);
void ewah_or_iterator_release(struct ewah_or_iterator *it);
void ewah_xor(
struct ewah_bitmap *ewah_i,
struct ewah_bitmap *ewah_j,

57
midx-write.c
View File

@ -647,16 +647,22 @@ static uint32_t *midx_pack_order(struct write_midx_context *ctx)
return pack_order;
}
static void write_midx_reverse_index(char *midx_name, unsigned char *midx_hash,
struct write_midx_context *ctx)
static void write_midx_reverse_index(struct write_midx_context *ctx,
const char *object_dir,
unsigned char *midx_hash)
{
struct strbuf buf = STRBUF_INIT;
char *tmp_file;
trace2_region_enter("midx", "write_midx_reverse_index", ctx->repo);
strbuf_addf(&buf, "%s-%s.rev", midx_name, hash_to_hex_algop(midx_hash,
ctx->repo->hash_algo));
if (ctx->incremental)
get_split_midx_filename_ext(ctx->repo->hash_algo, &buf,
object_dir, midx_hash,
MIDX_EXT_REV);
else
get_midx_filename_ext(ctx->repo->hash_algo, &buf, object_dir,
midx_hash, MIDX_EXT_REV);
tmp_file = write_rev_file_order(ctx->repo->hash_algo, NULL, ctx->pack_order,
ctx->entries_nr, midx_hash, WRITE_REV);
@ -829,22 +835,29 @@ static struct commit **find_commits_for_midx_bitmap(uint32_t *indexed_commits_nr
return cb.commits;
}
static int write_midx_bitmap(struct repository *r, const char *midx_name,
static int write_midx_bitmap(struct write_midx_context *ctx,
const char *object_dir,
const unsigned char *midx_hash,
struct packing_data *pdata,
struct commit **commits,
uint32_t commits_nr,
uint32_t *pack_order,
unsigned flags)
{
int ret, i;
uint16_t options = 0;
struct bitmap_writer writer;
struct pack_idx_entry **index;
char *bitmap_name = xstrfmt("%s-%s.bitmap", midx_name,
hash_to_hex_algop(midx_hash, r->hash_algo));
struct strbuf bitmap_name = STRBUF_INIT;
trace2_region_enter("midx", "write_midx_bitmap", r);
trace2_region_enter("midx", "write_midx_bitmap", ctx->repo);
if (ctx->incremental)
get_split_midx_filename_ext(ctx->repo->hash_algo, &bitmap_name,
object_dir, midx_hash,
MIDX_EXT_BITMAP);
else
get_midx_filename_ext(ctx->repo->hash_algo, &bitmap_name,
object_dir, midx_hash, MIDX_EXT_BITMAP);
if (flags & MIDX_WRITE_BITMAP_HASH_CACHE)
options |= BITMAP_OPT_HASH_CACHE;
@ -861,7 +874,8 @@ static int write_midx_bitmap(struct repository *r, const char *midx_name,
for (i = 0; i < pdata->nr_objects; i++)
index[i] = &pdata->objects[i].idx;
bitmap_writer_init(&writer, r, pdata);
bitmap_writer_init(&writer, ctx->repo, pdata,
ctx->incremental ? ctx->base_midx : NULL);
bitmap_writer_show_progress(&writer, flags & MIDX_PROGRESS);
bitmap_writer_build_type_index(&writer, index);
@ -879,7 +893,7 @@ static int write_midx_bitmap(struct repository *r, const char *midx_name,
* bitmap_writer_finish().
*/
for (i = 0; i < pdata->nr_objects; i++)
index[pack_order[i]] = &pdata->objects[i].idx;
index[ctx->pack_order[i]] = &pdata->objects[i].idx;
bitmap_writer_select_commits(&writer, commits, commits_nr);
ret = bitmap_writer_build(&writer);
@ -887,14 +901,14 @@ static int write_midx_bitmap(struct repository *r, const char *midx_name,
goto cleanup;
bitmap_writer_set_checksum(&writer, midx_hash);
bitmap_writer_finish(&writer, index, bitmap_name, options);
bitmap_writer_finish(&writer, index, bitmap_name.buf, options);
cleanup:
free(index);
free(bitmap_name);
strbuf_release(&bitmap_name);
bitmap_writer_free(&writer);
trace2_region_leave("midx", "write_midx_bitmap", r);
trace2_region_leave("midx", "write_midx_bitmap", ctx->repo);
return ret;
}
@ -1077,8 +1091,6 @@ static int write_midx_internal(struct repository *r, const char *object_dir,
ctx.repo = r;
ctx.incremental = !!(flags & MIDX_WRITE_INCREMENTAL);
if (ctx.incremental && (flags & MIDX_WRITE_BITMAP))
die(_("cannot write incremental MIDX with bitmap"));
if (ctx.incremental)
strbuf_addf(&midx_name,
@ -1119,6 +1131,13 @@ static int write_midx_internal(struct repository *r, const char *object_dir,
if (ctx.incremental) {
struct multi_pack_index *m = ctx.base_midx;
while (m) {
if (flags & MIDX_WRITE_BITMAP && load_midx_revindex(m)) {
error(_("could not load reverse index for MIDX %s"),
hash_to_hex_algop(get_midx_checksum(m),
m->repo->hash_algo));
result = 1;
goto cleanup;
}
ctx.num_multi_pack_indexes_before++;
m = m->base_midx;
}
@ -1387,7 +1406,7 @@ static int write_midx_internal(struct repository *r, const char *object_dir,
if (flags & MIDX_WRITE_REV_INDEX &&
git_env_bool("GIT_TEST_MIDX_WRITE_REV", 0))
write_midx_reverse_index(midx_name.buf, midx_hash, &ctx);
write_midx_reverse_index(&ctx, object_dir, midx_hash);
if (flags & MIDX_WRITE_BITMAP) {
struct packing_data pdata;
@ -1410,8 +1429,8 @@ static int write_midx_internal(struct repository *r, const char *object_dir,
FREE_AND_NULL(ctx.entries);
ctx.entries_nr = 0;
if (write_midx_bitmap(r, midx_name.buf, midx_hash, &pdata,
commits, commits_nr, ctx.pack_order,
if (write_midx_bitmap(&ctx, object_dir,
midx_hash, &pdata, commits, commits_nr,
flags) < 0) {
error(_("could not write multi-pack bitmap"));
result = 1;

65
pack-bitmap-write.c
View File

@ -26,6 +26,8 @@
#include "alloc.h"
#include "refs.h"
#include "strmap.h"
#include "midx.h"
#include "pack-revindex.h"
struct bitmapped_commit {
struct commit *commit;
@ -43,7 +45,8 @@ static inline int bitmap_writer_nr_selected_commits(struct bitmap_writer *writer
}
void bitmap_writer_init(struct bitmap_writer *writer, struct repository *r,
struct packing_data *pdata)
struct packing_data *pdata,
struct multi_pack_index *midx)
{
memset(writer, 0, sizeof(struct bitmap_writer));
if (writer->bitmaps)
@ -51,6 +54,7 @@ void bitmap_writer_init(struct bitmap_writer *writer, struct repository *r,
writer->bitmaps = kh_init_oid_map();
writer->pseudo_merge_commits = kh_init_oid_map();
writer->to_pack = pdata;
writer->midx = midx;
string_list_init_dup(&writer->pseudo_merge_groups);
@ -113,6 +117,11 @@ void bitmap_writer_build_type_index(struct bitmap_writer *writer,
struct pack_idx_entry **index)
{
uint32_t i;
uint32_t base_objects = 0;
if (writer->midx)
base_objects = writer->midx->num_objects +
writer->midx->num_objects_in_base;
writer->commits = ewah_new();
writer->trees = ewah_new();
@ -142,19 +151,19 @@ void bitmap_writer_build_type_index(struct bitmap_writer *writer,
switch (real_type) {
case OBJ_COMMIT:
ewah_set(writer->commits, i);
ewah_set(writer->commits, i + base_objects);
break;
case OBJ_TREE:
ewah_set(writer->trees, i);
ewah_set(writer->trees, i + base_objects);
break;
case OBJ_BLOB:
ewah_set(writer->blobs, i);
ewah_set(writer->blobs, i + base_objects);
break;
case OBJ_TAG:
ewah_set(writer->tags, i);
ewah_set(writer->tags, i + base_objects);
break;
default:
@ -207,19 +216,37 @@ void bitmap_writer_push_commit(struct bitmap_writer *writer,
static uint32_t find_object_pos(struct bitmap_writer *writer,
const struct object_id *oid, int *found)
{
struct object_entry *entry = packlist_find(writer->to_pack, oid);
struct object_entry *entry;
entry = packlist_find(writer->to_pack, oid);
if (entry) {
uint32_t base_objects = 0;
if (writer->midx)
base_objects = writer->midx->num_objects +
writer->midx->num_objects_in_base;
if (!entry) {
if (found)
*found = 0;
warning("Failed to write bitmap index. Packfile doesn't have full closure "
"(object %s is missing)", oid_to_hex(oid));
return 0;
*found = 1;
return oe_in_pack_pos(writer->to_pack, entry) + base_objects;
} else if (writer->midx) {
uint32_t at, pos;
if (!bsearch_midx(oid, writer->midx, &at))
goto missing;
if (midx_to_pack_pos(writer->midx, at, &pos) < 0)
goto missing;
if (found)
*found = 1;
return pos;
}
missing:
if (found)
*found = 1;
return oe_in_pack_pos(writer->to_pack, entry);
*found = 0;
warning("Failed to write bitmap index. Packfile doesn't have full closure "
"(object %s is missing)", oid_to_hex(oid));
return 0;
}
static void compute_xor_offsets(struct bitmap_writer *writer)
@ -586,7 +613,7 @@ int bitmap_writer_build(struct bitmap_writer *writer)
struct prio_queue queue = { compare_commits_by_gen_then_commit_date };
struct prio_queue tree_queue = { NULL };
struct bitmap_index *old_bitmap;
uint32_t *mapping;
uint32_t *mapping = NULL;
int closed = 1; /* until proven otherwise */
if (writer->show_progress)
@ -1021,7 +1048,7 @@ void bitmap_writer_finish(struct bitmap_writer *writer,
struct strbuf tmp_file = STRBUF_INIT;
struct hashfile *f;
off_t *offsets = NULL;
uint32_t i;
uint32_t i, base_objects;
struct bitmap_disk_header header;
@ -1047,6 +1074,12 @@ void bitmap_writer_finish(struct bitmap_writer *writer,
if (options & BITMAP_OPT_LOOKUP_TABLE)
CALLOC_ARRAY(offsets, writer->to_pack->nr_objects);
if (writer->midx)
base_objects = writer->midx->num_objects +
writer->midx->num_objects_in_base;
else
base_objects = 0;
for (i = 0; i < bitmap_writer_nr_selected_commits(writer); i++) {
struct bitmapped_commit *stored = &writer->selected[i];
int commit_pos = oid_pos(&stored->commit->object.oid, index,
@ -1055,7 +1088,7 @@ void bitmap_writer_finish(struct bitmap_writer *writer,
if (commit_pos < 0)
BUG(_("trying to write commit not in index"));
stored->commit_pos = commit_pos;
stored->commit_pos = commit_pos + base_objects;
}
write_selected_commits_v1(writer, f, offsets);

344
pack-bitmap.c
View File

@ -54,6 +54,16 @@ struct bitmap_index {
struct packed_git *pack;
struct multi_pack_index *midx;
/*
* If using a multi-pack index chain, 'base' points to the
* bitmap index corresponding to this bitmap's midx->base_midx.
*
* base_nr indicates how many layers precede this one, and is
* zero when base is NULL.
*/
struct bitmap_index *base;
uint32_t base_nr;
/* mmapped buffer of the whole bitmap index */
unsigned char *map;
size_t map_size; /* size of the mmaped buffer */
@ -71,6 +81,23 @@ struct bitmap_index {
struct ewah_bitmap *blobs;
struct ewah_bitmap *tags;
/*
* Type index arrays when this bitmap is associated with an
* incremental multi-pack index chain.
*
* If n is the number of unique layers in the MIDX chain, then
* commits_all[n-1] is this structs 'commits' field,
* commits_all[n-2] is the commits field of this bitmap's
* 'base', and so on.
*
* When associated either with a non-incremental MIDX or a
* single packfile, these arrays each contain a single element.
*/
struct ewah_bitmap **commits_all;
struct ewah_bitmap **trees_all;
struct ewah_bitmap **blobs_all;
struct ewah_bitmap **tags_all;
/* Map from object ID -> `stored_bitmap` for all the bitmapped commits */
kh_oid_map_t *bitmaps;
@ -170,6 +197,15 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)
return read_bitmap(index->map, index->map_size, &index->map_pos);
}
static uint32_t bitmap_num_objects_total(struct bitmap_index *index)
{
if (index->midx) {
struct multi_pack_index *m = index->midx;
return m->num_objects + m->num_objects_in_base;
}
return index->pack->num_objects;
}
static uint32_t bitmap_num_objects(struct bitmap_index *index)
{
if (index->midx)
@ -377,8 +413,15 @@ static int load_bitmap_entries_v1(struct bitmap_index *index)
char *midx_bitmap_filename(struct multi_pack_index *midx)
{
struct strbuf buf = STRBUF_INIT;
get_midx_filename_ext(midx->repo->hash_algo, &buf, midx->object_dir,
get_midx_checksum(midx), MIDX_EXT_BITMAP);
if (midx->has_chain)
get_split_midx_filename_ext(midx->repo->hash_algo, &buf,
midx->object_dir,
get_midx_checksum(midx),
MIDX_EXT_BITMAP);
else
get_midx_filename_ext(midx->repo->hash_algo, &buf,
midx->object_dir, get_midx_checksum(midx),
MIDX_EXT_BITMAP);
return strbuf_detach(&buf, NULL);
}
@ -445,16 +488,21 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,
goto cleanup;
}
for (i = 0; i < bitmap_git->midx->num_packs; i++) {
if (prepare_midx_pack(bitmap_repo(bitmap_git),
bitmap_git->midx,
i)) {
for (i = 0; i < bitmap_git->midx->num_packs + bitmap_git->midx->num_packs_in_base; i++) {
if (prepare_midx_pack(bitmap_repo(bitmap_git), bitmap_git->midx, i)) {
warning(_("could not open pack %s"),
bitmap_git->midx->pack_names[i]);
goto cleanup;
}
}
if (midx->base_midx) {
bitmap_git->base = prepare_midx_bitmap_git(midx->base_midx);
bitmap_git->base_nr = bitmap_git->base->base_nr + 1;
} else {
bitmap_git->base_nr = 0;
}
return 0;
cleanup:
@ -506,6 +554,7 @@ static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git
bitmap_git->map_size = xsize_t(st.st_size);
bitmap_git->map = xmmap(NULL, bitmap_git->map_size, PROT_READ, MAP_PRIVATE, fd, 0);
bitmap_git->map_pos = 0;
bitmap_git->base_nr = 0;
close(fd);
if (load_bitmap_header(bitmap_git) < 0) {
@ -525,8 +574,7 @@ static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git
static int load_reverse_index(struct repository *r, struct bitmap_index *bitmap_git)
{
if (bitmap_is_midx(bitmap_git)) {
uint32_t i;
int ret;
struct multi_pack_index *m;
/*
* The multi-pack-index's .rev file is already loaded via
@ -535,17 +583,47 @@ static int load_reverse_index(struct repository *r, struct bitmap_index *bitmap_
* But we still need to open the individual pack .rev files,
* since we will need to make use of them in pack-objects.
*/
for (i = 0; i < bitmap_git->midx->num_packs; i++) {
ret = load_pack_revindex(r, bitmap_git->midx->packs[i]);
if (ret)
return ret;
for (m = bitmap_git->midx; m; m = m->base_midx) {
uint32_t i;
int ret;
for (i = 0; i < m->num_packs; i++) {
ret = load_pack_revindex(r, m->packs[i]);
if (ret)
return ret;
}
}
return 0;
}
return load_pack_revindex(r, bitmap_git->pack);
}
static int load_bitmap(struct repository *r, struct bitmap_index *bitmap_git)
static void load_all_type_bitmaps(struct bitmap_index *bitmap_git)
{
struct bitmap_index *curr = bitmap_git;
size_t i = bitmap_git->base_nr;
ALLOC_ARRAY(bitmap_git->commits_all, bitmap_git->base_nr + 1);
ALLOC_ARRAY(bitmap_git->trees_all, bitmap_git->base_nr + 1);
ALLOC_ARRAY(bitmap_git->blobs_all, bitmap_git->base_nr + 1);
ALLOC_ARRAY(bitmap_git->tags_all, bitmap_git->base_nr + 1);
while (curr) {
bitmap_git->commits_all[i] = curr->commits;
bitmap_git->trees_all[i] = curr->trees;
bitmap_git->blobs_all[i] = curr->blobs;
bitmap_git->tags_all[i] = curr->tags;
curr = curr->base;
if (curr && !i)
BUG("unexpected number of bitmap layers, expected %"PRIu32,
bitmap_git->base_nr + 1);
i -= 1;
}
}
static int load_bitmap(struct repository *r, struct bitmap_index *bitmap_git,
int recursing)
{
assert(bitmap_git->map);
@ -564,6 +642,16 @@ static int load_bitmap(struct repository *r, struct bitmap_index *bitmap_git)
if (!bitmap_git->table_lookup && load_bitmap_entries_v1(bitmap_git) < 0)
goto failed;
if (bitmap_git->base) {
if (!bitmap_is_midx(bitmap_git))
BUG("non-MIDX bitmap has non-NULL base bitmap index");
if (load_bitmap(r, bitmap_git->base, 1) < 0)
goto failed;
}
if (!recursing)
load_all_type_bitmaps(bitmap_git);
return 0;
failed:
@ -639,7 +727,7 @@ struct bitmap_index *prepare_bitmap_git(struct repository *r)
{
struct bitmap_index *bitmap_git = xcalloc(1, sizeof(*bitmap_git));
if (!open_bitmap(r, bitmap_git) && !load_bitmap(r, bitmap_git))
if (!open_bitmap(r, bitmap_git) && !load_bitmap(r, bitmap_git, 0))
return bitmap_git;
free_bitmap_index(bitmap_git);
@ -648,10 +736,9 @@ struct bitmap_index *prepare_bitmap_git(struct repository *r)
struct bitmap_index *prepare_midx_bitmap_git(struct multi_pack_index *midx)
{
struct repository *r = midx->repo;
struct bitmap_index *bitmap_git = xcalloc(1, sizeof(*bitmap_git));
if (!open_midx_bitmap_1(bitmap_git, midx) && !load_bitmap(r, bitmap_git))
if (!open_midx_bitmap_1(bitmap_git, midx))
return bitmap_git;
free_bitmap_index(bitmap_git);
@ -896,26 +983,42 @@ corrupt:
return NULL;
}
struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,
struct commit *commit)
static struct ewah_bitmap *find_bitmap_for_commit(struct bitmap_index *bitmap_git,
struct commit *commit,
struct bitmap_index **found)
{
khiter_t hash_pos = kh_get_oid_map(bitmap_git->bitmaps,
commit->object.oid);
khiter_t hash_pos;
if (!bitmap_git)
return NULL;
hash_pos = kh_get_oid_map(bitmap_git->bitmaps, commit->object.oid);
if (hash_pos >= kh_end(bitmap_git->bitmaps)) {
struct stored_bitmap *bitmap = NULL;
if (!bitmap_git->table_lookup)
return NULL;
return find_bitmap_for_commit(bitmap_git->base, commit,
found);
/* this is a fairly hot codepath - no trace2_region please */
/* NEEDSWORK: cache misses aren't recorded */
bitmap = lazy_bitmap_for_commit(bitmap_git, commit);
if (!bitmap)
return NULL;
return find_bitmap_for_commit(bitmap_git->base, commit,
found);
if (found)
*found = bitmap_git;
return lookup_stored_bitmap(bitmap);
}
if (found)
*found = bitmap_git;
return lookup_stored_bitmap(kh_value(bitmap_git->bitmaps, hash_pos));
}
struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,
struct commit *commit)
{
return find_bitmap_for_commit(bitmap_git, commit, NULL);
}
static inline int bitmap_position_extended(struct bitmap_index *bitmap_git,
const struct object_id *oid)
{
@ -924,7 +1027,7 @@ static inline int bitmap_position_extended(struct bitmap_index *bitmap_git,
if (pos < kh_end(positions)) {
int bitmap_pos = kh_value(positions, pos);
return bitmap_pos + bitmap_num_objects(bitmap_git);
return bitmap_pos + bitmap_num_objects_total(bitmap_git);
}
return -1;
@ -992,7 +1095,7 @@ static int ext_index_add_object(struct bitmap_index *bitmap_git,
bitmap_pos = kh_value(eindex->positions, hash_pos);
}
return bitmap_pos + bitmap_num_objects(bitmap_git);
return bitmap_pos + bitmap_num_objects_total(bitmap_git);
}
struct bitmap_show_data {
@ -1024,10 +1127,15 @@ static unsigned apply_pseudo_merges_for_commit_1(struct bitmap_index *bitmap_git
struct commit *commit,
uint32_t commit_pos)
{
int ret;
struct bitmap_index *curr = bitmap_git;
int ret = 0;
ret = apply_pseudo_merges_for_commit(&bitmap_git->pseudo_merges,
result, commit, commit_pos);
while (curr) {
ret += apply_pseudo_merges_for_commit(&curr->pseudo_merges,
result, commit,
commit_pos);
curr = curr->base;
}
if (ret)
pseudo_merges_satisfied_nr += ret;
@ -1342,11 +1450,17 @@ struct ewah_bitmap *pseudo_merge_bitmap_for_commit(struct bitmap_index *bitmap_g
if (pos < 0 || pos >= bitmap_num_objects(bitmap_git))
goto done;
/*
* Use bitmap-relative positions instead of offsetting
* by bitmap_git->num_objects_in_base because we use
* this to find a match in pseudo_merge_for_parents(),
* and pseudo-merge groups cannot span multiple bitmap
* layers.
*/
bitmap_set(parents, pos);
}
match = pseudo_merge_for_parents(&bitmap_git->pseudo_merges,
parents);
match = pseudo_merge_for_parents(&bitmap_git->pseudo_merges, parents);
done:
bitmap_free(parents);
@ -1500,7 +1614,9 @@ static void show_extended_objects(struct bitmap_index *bitmap_git,
for (i = 0; i < eindex->count; ++i) {
struct object *obj;
if (!bitmap_get(objects, st_add(bitmap_num_objects(bitmap_git), i)))
if (!bitmap_get(objects,
st_add(bitmap_num_objects_total(bitmap_git),
i)))
continue;
obj = eindex->objects[i];
@ -1513,25 +1629,29 @@ static void show_extended_objects(struct bitmap_index *bitmap_git,
}
}
static void init_type_iterator(struct ewah_iterator *it,
static void init_type_iterator(struct ewah_or_iterator *it,
struct bitmap_index *bitmap_git,
enum object_type type)
{
switch (type) {
case OBJ_COMMIT:
ewah_iterator_init(it, bitmap_git->commits);
ewah_or_iterator_init(it, bitmap_git->commits_all,
bitmap_git->base_nr + 1);
break;
case OBJ_TREE:
ewah_iterator_init(it, bitmap_git->trees);
ewah_or_iterator_init(it, bitmap_git->trees_all,
bitmap_git->base_nr + 1);
break;
case OBJ_BLOB:
ewah_iterator_init(it, bitmap_git->blobs);
ewah_or_iterator_init(it, bitmap_git->blobs_all,
bitmap_git->base_nr + 1);
break;
case OBJ_TAG:
ewah_iterator_init(it, bitmap_git->tags);
ewah_or_iterator_init(it, bitmap_git->tags_all,
bitmap_git->base_nr + 1);
break;
default:
@ -1548,7 +1668,7 @@ static void show_objects_for_type(
size_t i = 0;
uint32_t offset;
struct ewah_iterator it;
struct ewah_or_iterator it;
eword_t filter;
struct bitmap *objects = bitmap_git->result;
@ -1556,7 +1676,7 @@ static void show_objects_for_type(
init_type_iterator(&it, bitmap_git, object_type);
for (i = 0; i < objects->word_alloc &&
ewah_iterator_next(&filter, &it); i++) {
ewah_or_iterator_next(&filter, &it); i++) {
eword_t word = objects->words[i] & filter;
size_t pos = (i * BITS_IN_EWORD);
@ -1583,7 +1703,7 @@ static void show_objects_for_type(
nth_midxed_object_oid(&oid, m, index_pos);
pack_id = nth_midxed_pack_int_id(m, index_pos);
pack = bitmap_git->midx->packs[pack_id];
pack = nth_midxed_pack(bitmap_git->midx, pack_id);
} else {
index_pos = pack_pos_to_index(bitmap_git->pack, pos + offset);
ofs = pack_pos_to_offset(bitmap_git->pack, pos + offset);
@ -1598,6 +1718,8 @@ static void show_objects_for_type(
show_reach(&oid, object_type, 0, hash, pack, ofs);
}
}
ewah_or_iterator_release(&it);
}
static int in_bitmapped_pack(struct bitmap_index *bitmap_git,
@ -1649,7 +1771,7 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,
{
struct eindex *eindex = &bitmap_git->ext_index;
struct bitmap *tips;
struct ewah_iterator it;
struct ewah_or_iterator it;
eword_t mask;
uint32_t i;
@ -1666,7 +1788,7 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,
* packfile.
*/
for (i = 0, init_type_iterator(&it, bitmap_git, type);
i < to_filter->word_alloc && ewah_iterator_next(&mask, &it);
i < to_filter->word_alloc && ewah_or_iterator_next(&mask, &it);
i++) {
if (i < tips->word_alloc)
mask &= ~tips->words[i];
@ -1679,13 +1801,14 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,
* them individually.
*/
for (i = 0; i < eindex->count; i++) {
size_t pos = st_add(i, bitmap_num_objects(bitmap_git));
size_t pos = st_add(i, bitmap_num_objects_total(bitmap_git));
if (eindex->objects[i]->type == type &&
bitmap_get(to_filter, pos) &&
!bitmap_get(tips, pos))
bitmap_unset(to_filter, pos);
}
ewah_or_iterator_release(&it);
bitmap_free(tips);
}
@ -1705,7 +1828,7 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,
oi.sizep = &size;
if (pos < bitmap_num_objects(bitmap_git)) {
if (pos < bitmap_num_objects_total(bitmap_git)) {
struct packed_git *pack;
off_t ofs;
@ -1713,7 +1836,7 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,
uint32_t midx_pos = pack_pos_to_midx(bitmap_git->midx, pos);
uint32_t pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos);
pack = bitmap_git->midx->packs[pack_id];
pack = nth_midxed_pack(bitmap_git->midx, pack_id);
ofs = nth_midxed_offset(bitmap_git->midx, midx_pos);
} else {
pack = bitmap_git->pack;
@ -1728,8 +1851,9 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,
die(_("unable to get size of %s"), oid_to_hex(&oid));
}
} else {
size_t eindex_pos = pos - bitmap_num_objects_total(bitmap_git);
struct eindex *eindex = &bitmap_git->ext_index;
struct object *obj = eindex->objects[pos - bitmap_num_objects(bitmap_git)];
struct object *obj = eindex->objects[eindex_pos];
if (oid_object_info_extended(bitmap_repo(bitmap_git), &obj->oid,
&oi, 0) < 0)
die(_("unable to get size of %s"), oid_to_hex(&obj->oid));
@ -1745,14 +1869,14 @@ static void filter_bitmap_blob_limit(struct bitmap_index *bitmap_git,
{
struct eindex *eindex = &bitmap_git->ext_index;
struct bitmap *tips;
struct ewah_iterator it;
struct ewah_or_iterator it;
eword_t mask;
uint32_t i;
tips = find_tip_objects(bitmap_git, tip_objects, OBJ_BLOB);
for (i = 0, init_type_iterator(&it, bitmap_git, OBJ_BLOB);
i < to_filter->word_alloc && ewah_iterator_next(&mask, &it);
i < to_filter->word_alloc && ewah_or_iterator_next(&mask, &it);
i++) {
eword_t word = to_filter->words[i] & mask;
unsigned offset;
@ -1780,6 +1904,7 @@ static void filter_bitmap_blob_limit(struct bitmap_index *bitmap_git,
bitmap_unset(to_filter, pos);
}
ewah_or_iterator_release(&it);
bitmap_free(tips);
}
@ -1882,7 +2007,7 @@ static void filter_packed_objects_from_bitmap(struct bitmap_index *bitmap_git,
uint32_t objects_nr;
size_t i, pos;
objects_nr = bitmap_num_objects(bitmap_git);
objects_nr = bitmap_num_objects_total(bitmap_git);
pos = objects_nr / BITS_IN_EWORD;
if (pos > result->word_alloc)
@ -1980,7 +2105,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,
* from disk. this is the point of no return; after this the rev_list
* becomes invalidated and we must perform the revwalk through bitmaps
*/
if (load_bitmap(revs->repo, bitmap_git) < 0)
if (load_bitmap(revs->repo, bitmap_git, 0) < 0)
goto cleanup;
if (!use_boundary_traversal)
@ -2281,7 +2406,8 @@ void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,
multi_pack_reuse = 0;
if (multi_pack_reuse) {
for (i = 0; i < bitmap_git->midx->num_packs; i++) {
struct multi_pack_index *m = bitmap_git->midx;
for (i = 0; i < m->num_packs + m->num_packs_in_base; i++) {
struct bitmapped_pack pack;
if (nth_bitmapped_pack(r, bitmap_git->midx, &pack, i) < 0) {
warning(_("unable to load pack: '%s', disabling pack-reuse"),
@ -2307,14 +2433,18 @@ void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,
uint32_t pack_int_id;
if (bitmap_is_midx(bitmap_git)) {
struct multi_pack_index *m = bitmap_git->midx;
uint32_t preferred_pack_pos;
if (midx_preferred_pack(bitmap_git->midx, &preferred_pack_pos) < 0) {
while (m->base_midx)
m = m->base_midx;
if (midx_preferred_pack(m, &preferred_pack_pos) < 0) {
warning(_("unable to compute preferred pack, disabling pack-reuse"));
return;
}
pack = bitmap_git->midx->packs[preferred_pack_pos];
pack = nth_midxed_pack(m, preferred_pack_pos);
pack_int_id = preferred_pack_pos;
} else {
pack = bitmap_git->pack;
@ -2406,12 +2536,12 @@ static uint32_t count_object_type(struct bitmap_index *bitmap_git,
struct eindex *eindex = &bitmap_git->ext_index;
uint32_t i = 0, count = 0;
struct ewah_iterator it;
struct ewah_or_iterator it;
eword_t filter;
init_type_iterator(&it, bitmap_git, type);
while (i < objects->word_alloc && ewah_iterator_next(&filter, &it)) {
while (i < objects->word_alloc && ewah_or_iterator_next(&filter, &it)) {
eword_t word = objects->words[i++] & filter;
count += ewah_bit_popcount64(word);
}
@ -2419,10 +2549,12 @@ static uint32_t count_object_type(struct bitmap_index *bitmap_git,
for (i = 0; i < eindex->count; ++i) {
if (eindex->objects[i]->type == type &&
bitmap_get(objects,
st_add(bitmap_num_objects(bitmap_git), i)))
st_add(bitmap_num_objects_total(bitmap_git), i)))
count++;
}
ewah_or_iterator_release(&it);
return count;
}
@ -2454,6 +2586,8 @@ struct bitmap_test_data {
struct bitmap *tags;
struct progress *prg;
size_t seen;
struct bitmap_test_data *base_tdata;
};
static void test_bitmap_type(struct bitmap_test_data *tdata,
@ -2462,6 +2596,11 @@ static void test_bitmap_type(struct bitmap_test_data *tdata,
enum object_type bitmap_type = OBJ_NONE;
int bitmaps_nr = 0;
if (bitmap_is_midx(tdata->bitmap_git)) {
while (pos < tdata->bitmap_git->midx->num_objects_in_base)
tdata = tdata->base_tdata;
}
if (bitmap_get(tdata->commits, pos)) {
bitmap_type = OBJ_COMMIT;
bitmaps_nr++;
@ -2525,13 +2664,57 @@ static void test_show_commit(struct commit *commit, void *data)
display_progress(tdata->prg, ++tdata->seen);
}
static uint32_t bitmap_total_entry_count(struct bitmap_index *bitmap_git)
{
uint32_t total = 0;
do {
total = st_add(total, bitmap_git->entry_count);
bitmap_git = bitmap_git->base;
} while (bitmap_git);
return total;
}
static void bitmap_test_data_prepare(struct bitmap_test_data *tdata,
struct bitmap_index *bitmap_git)
{
memset(tdata, 0, sizeof(struct bitmap_test_data));
tdata->bitmap_git = bitmap_git;
tdata->base = bitmap_new();
tdata->commits = ewah_to_bitmap(bitmap_git->commits);
tdata->trees = ewah_to_bitmap(bitmap_git->trees);
tdata->blobs = ewah_to_bitmap(bitmap_git->blobs);
tdata->tags = ewah_to_bitmap(bitmap_git->tags);
if (bitmap_git->base) {
tdata->base_tdata = xmalloc(sizeof(struct bitmap_test_data));
bitmap_test_data_prepare(tdata->base_tdata, bitmap_git->base);
}
}
static void bitmap_test_data_release(struct bitmap_test_data *tdata)
{
if (!tdata)
return;
bitmap_test_data_release(tdata->base_tdata);
free(tdata->base_tdata);
bitmap_free(tdata->base);
bitmap_free(tdata->commits);
bitmap_free(tdata->trees);
bitmap_free(tdata->blobs);
bitmap_free(tdata->tags);
}
void test_bitmap_walk(struct rev_info *revs)
{
struct object *root;
struct bitmap *result = NULL;
size_t result_popcnt;
struct bitmap_test_data tdata;
struct bitmap_index *bitmap_git;
struct bitmap_index *bitmap_git, *found;
struct ewah_bitmap *bm;
if (!(bitmap_git = prepare_bitmap_git(revs->repo)))
@ -2540,17 +2723,28 @@ void test_bitmap_walk(struct rev_info *revs)
if (revs->pending.nr != 1)
die(_("you must specify exactly one commit to test"));
fprintf_ln(stderr, "Bitmap v%d test (%d entries%s)",
fprintf_ln(stderr, "Bitmap v%d test (%d entries%s, %d total)",
bitmap_git->version,
bitmap_git->entry_count,
bitmap_git->table_lookup ? "" : " loaded");
bitmap_git->table_lookup ? "" : " loaded",
bitmap_total_entry_count(bitmap_git));
root = revs->pending.objects[0].item;
bm = bitmap_for_commit(bitmap_git, (struct commit *)root);
bm = find_bitmap_for_commit(bitmap_git, (struct commit *)root, &found);
if (bm) {
fprintf_ln(stderr, "Found bitmap for '%s'. %d bits / %08x checksum",
oid_to_hex(&root->oid), (int)bm->bit_size, ewah_checksum(bm));
oid_to_hex(&root->oid),
(int)bm->bit_size, ewah_checksum(bm));
if (bitmap_is_midx(found))
fprintf_ln(stderr, "Located via MIDX '%s'.",
hash_to_hex_algop(get_midx_checksum(found->midx),
revs->repo->hash_algo));
else
fprintf_ln(stderr, "Located via pack '%s'.",
hash_to_hex_algop(found->pack->hash,
revs->repo->hash_algo));
result = ewah_to_bitmap(bm);
}
@ -2567,16 +2761,10 @@ void test_bitmap_walk(struct rev_info *revs)
if (prepare_revision_walk(revs))
die(_("revision walk setup failed"));
tdata.bitmap_git = bitmap_git;
tdata.base = bitmap_new();
tdata.commits = ewah_to_bitmap(bitmap_git->commits);
tdata.trees = ewah_to_bitmap(bitmap_git->trees);
tdata.blobs = ewah_to_bitmap(bitmap_git->blobs);
tdata.tags = ewah_to_bitmap(bitmap_git->tags);
bitmap_test_data_prepare(&tdata, bitmap_git);
tdata.prg = start_progress(revs->repo,
"Verifying bitmap entries",
result_popcnt);
tdata.seen = 0;
traverse_commit_list(revs, &test_show_commit, &test_show_object, &tdata);
@ -2588,11 +2776,7 @@ void test_bitmap_walk(struct rev_info *revs)
die(_("mismatch in bitmap results"));
bitmap_free(result);
bitmap_free(tdata.base);
bitmap_free(tdata.commits);
bitmap_free(tdata.trees);
bitmap_free(tdata.blobs);
bitmap_free(tdata.tags);
bitmap_test_data_release(&tdata);
free_bitmap_index(bitmap_git);
}
@ -2820,7 +3004,7 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,
BUG("rebuild_existing_bitmaps: missing required rev-cache "
"extension");
num_objects = bitmap_num_objects(bitmap_git);
num_objects = bitmap_num_objects_total(bitmap_git);
CALLOC_ARRAY(reposition, num_objects);
for (i = 0; i < num_objects; ++i) {
@ -2856,6 +3040,10 @@ void free_bitmap_index(struct bitmap_index *b)
ewah_pool_free(b->trees);
ewah_pool_free(b->blobs);
ewah_pool_free(b->tags);
free(b->commits_all);
free(b->trees_all);
free(b->blobs_all);
free(b->tags_all);
if (b->bitmaps) {
struct stored_bitmap *sb;
kh_foreach_value(b->bitmaps, sb, {
@ -2883,6 +3071,7 @@ void free_bitmap_index(struct bitmap_index *b)
close_midx_revindex(b->midx);
}
free_pseudo_merge_map(&b->pseudo_merges);
free_bitmap_index(b->base);
free(b);
}
@ -2898,13 +3087,13 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,
{
struct bitmap *result = bitmap_git->result;
off_t total = 0;
struct ewah_iterator it;
struct ewah_or_iterator it;
eword_t filter;
size_t i;
init_type_iterator(&it, bitmap_git, object_type);
for (i = 0; i < result->word_alloc &&
ewah_iterator_next(&filter, &it); i++) {
ewah_or_iterator_next(&filter, &it); i++) {
eword_t word = result->words[i] & filter;
size_t base = (i * BITS_IN_EWORD);
unsigned offset;
@ -2924,7 +3113,7 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,
off_t offset = nth_midxed_offset(bitmap_git->midx, midx_pos);
uint32_t pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos);
struct packed_git *pack = bitmap_git->midx->packs[pack_id];
struct packed_git *pack = nth_midxed_pack(bitmap_git->midx, pack_id);
if (offset_to_pack_pos(pack, offset, &pack_pos) < 0) {
struct object_id oid;
@ -2945,6 +3134,8 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git,
}
}
ewah_or_iterator_release(&it);
return total;
}
@ -2963,7 +3154,8 @@ static off_t get_disk_usage_for_extended(struct bitmap_index *bitmap_git)
struct object *obj = eindex->objects[i];
if (!bitmap_get(result,
st_add(bitmap_num_objects(bitmap_git), i)))
st_add(bitmap_num_objects_total(bitmap_git),
i)))
continue;
if (oid_object_info_extended(bitmap_repo(bitmap_git), &obj->oid,

4
pack-bitmap.h
View File

@ -111,6 +111,7 @@ struct bitmap_writer {
kh_oid_map_t *bitmaps;
struct packing_data *to_pack;
struct multi_pack_index *midx; /* if appending to a MIDX chain */
struct bitmapped_commit *selected;
unsigned int selected_nr, selected_alloc;
@ -125,7 +126,8 @@ struct bitmap_writer {
};
void bitmap_writer_init(struct bitmap_writer *writer, struct repository *r,
struct packing_data *pdata);
struct packing_data *pdata,
struct multi_pack_index *midx);
void bitmap_writer_show_progress(struct bitmap_writer *writer, int show);
void bitmap_writer_set_checksum(struct bitmap_writer *writer,
const unsigned char *sha1);

34
pack-revindex.c
View File

@ -383,8 +383,14 @@ int load_midx_revindex(struct multi_pack_index *m)
trace2_data_string("load_midx_revindex", the_repository,
"source", "rev");
get_midx_filename_ext(m->repo->hash_algo, &revindex_name, m->object_dir,
get_midx_checksum(m), MIDX_EXT_REV);
if (m->has_chain)
get_split_midx_filename_ext(m->repo->hash_algo, &revindex_name,
m->object_dir, get_midx_checksum(m),
MIDX_EXT_REV);
else
get_midx_filename_ext(m->repo->hash_algo, &revindex_name,
m->object_dir, get_midx_checksum(m),
MIDX_EXT_REV);
ret = load_revindex_from_disk(revindex_name.buf,
m->num_objects,
@ -471,11 +477,15 @@ off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos)
uint32_t pack_pos_to_midx(struct multi_pack_index *m, uint32_t pos)
{
while (m && pos < m->num_objects_in_base)
m = m->base_midx;
if (!m)
BUG("NULL multi-pack-index for object position: %"PRIu32, pos);
if (!m->revindex_data)
BUG("pack_pos_to_midx: reverse index not yet loaded");
if (m->num_objects <= pos)
if (m->num_objects + m->num_objects_in_base <= pos)
BUG("pack_pos_to_midx: out-of-bounds object at %"PRIu32, pos);
return get_be32(m->revindex_data + pos);
return get_be32(m->revindex_data + pos - m->num_objects_in_base);
}
struct midx_pack_key {
@ -491,7 +501,8 @@ static int midx_pack_order_cmp(const void *va, const void *vb)
const struct midx_pack_key *key = va;
struct multi_pack_index *midx = key->midx;
uint32_t versus = pack_pos_to_midx(midx, (uint32_t*)vb - (const uint32_t *)midx->revindex_data);
size_t pos = (uint32_t *)vb - (const uint32_t *)midx->revindex_data;
uint32_t versus = pack_pos_to_midx(midx, pos + midx->num_objects_in_base);
uint32_t versus_pack = nth_midxed_pack_int_id(midx, versus);
off_t versus_offset;
@ -529,9 +540,9 @@ static int midx_key_to_pack_pos(struct multi_pack_index *m,
{
uint32_t *found;
if (key->pack >= m->num_packs)
if (key->pack >= m->num_packs + m->num_packs_in_base)
BUG("MIDX pack lookup out of bounds (%"PRIu32" >= %"PRIu32")",
key->pack, m->num_packs);
key->pack, m->num_packs + m->num_packs_in_base);
/*
* The preferred pack sorts first, so determine its identifier by
* looking at the first object in pseudo-pack order.
@ -551,7 +562,8 @@ static int midx_key_to_pack_pos(struct multi_pack_index *m,
if (!found)
return -1;
*pos = found - m->revindex_data;
*pos = (found - m->revindex_data) + m->num_objects_in_base;
return 0;
}
@ -559,9 +571,13 @@ int midx_to_pack_pos(struct multi_pack_index *m, uint32_t at, uint32_t *pos)
{
struct midx_pack_key key;
while (m && at < m->num_objects_in_base)
m = m->base_midx;
if (!m)
BUG("NULL multi-pack-index for object position: %"PRIu32, at);
if (!m->revindex_data)
BUG("midx_to_pack_pos: reverse index not yet loaded");
if (m->num_objects <= at)
if (m->num_objects + m->num_objects_in_base <= at)
BUG("midx_to_pack_pos: out-of-bounds object at %"PRIu32, at);
key.pack = nth_midxed_pack_int_id(m, at);

87
t/t5334-incremental-multi-pack-index.sh
View File

@ -44,4 +44,91 @@ test_expect_success 'convert incremental to non-incremental' '
compare_results_with_midx 'non-incremental MIDX conversion'
write_midx_layer () {
n=1
if test -f $midx_chain
then
n="$(($(wc -l <$midx_chain) + 1))"
fi
for i in 1 2
do
test_commit $n.$i &&
git repack -d || return 1
done &&
git multi-pack-index write --bitmap --incremental
}
test_expect_success 'write initial MIDX layer' '
git repack -ad &&
write_midx_layer
'
test_expect_success 'read bitmap from first MIDX layer' '
git rev-list --test-bitmap 1.2
'
test_expect_success 'write another MIDX layer' '
write_midx_layer
'
test_expect_success 'midx verify with multiple layers' '
test_path_is_file "$midx_chain" &&
test_line_count = 2 "$midx_chain" &&
git multi-pack-index verify
'
test_expect_success 'read bitmap from second MIDX layer' '
git rev-list --test-bitmap 2.2
'
test_expect_success 'read earlier bitmap from second MIDX layer' '
git rev-list --test-bitmap 1.2
'
test_expect_success 'show object from first pack' '
git cat-file -p 1.1
'
test_expect_success 'show object from second pack' '
git cat-file -p 2.2
'
for reuse in false single multi
do
test_expect_success "full clone (pack.allowPackReuse=$reuse)" '
rm -fr clone.git &&
git config pack.allowPackReuse $reuse &&
git clone --no-local --bare . clone.git
'
done
test_expect_success 'relink existing MIDX layer' '
rm -fr "$midxdir" &&
GIT_TEST_MIDX_WRITE_REV=1 git multi-pack-index write --bitmap &&
midx_hash="$(test-tool read-midx --checksum $objdir)" &&
test_path_is_file "$packdir/multi-pack-index" &&
test_path_is_file "$packdir/multi-pack-index-$midx_hash.bitmap" &&
test_path_is_file "$packdir/multi-pack-index-$midx_hash.rev" &&
test_commit another &&
git repack -d &&
git multi-pack-index write --bitmap --incremental &&
test_path_is_missing "$packdir/multi-pack-index" &&
test_path_is_missing "$packdir/multi-pack-index-$midx_hash.bitmap" &&
test_path_is_missing "$packdir/multi-pack-index-$midx_hash.rev" &&
test_path_is_file "$midxdir/multi-pack-index-$midx_hash.midx" &&
test_path_is_file "$midxdir/multi-pack-index-$midx_hash.bitmap" &&
test_path_is_file "$midxdir/multi-pack-index-$midx_hash.rev" &&
test_line_count = 2 "$midx_chain"
'
test_done