-
Notifications
You must be signed in to change notification settings - Fork 97
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
gvfs-helper: create tool to fetch objects using the GVFS Protocol #191
gvfs-helper: create tool to fetch objects using the GVFS Protocol #191
Conversation
65be96a
to
05f71f0
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is really coming together!
My only big comment is that we'll want to spin out many of the TODOs as issues so we remember to follow up later.
* later rename it to a proper .pack and run "git index-pack" on | ||
* it to create the corresponding .idx file. | ||
* | ||
* TODO I would rather to just stream the packfile directly into |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is an important TODO to handle in a follow-up. Let's create an issue for it.
(long)tv.tv_usec); | ||
} | ||
|
||
// TODO should this be in the "<ODB>/pack/tempPacks/" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, the tempPacks
dir would more closely match the behavior of scalar prefetch --folders
and we would avoid issues with hardware failure before a full flush during indexing.
// TODO consider capturing stdout from index-pack because | ||
// TODO it will contain the SHA of the packfile and we can | ||
// TODO (should?) add it to the .pack and .idx pathnames | ||
// TODO when we install them. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be best to add it to the names, if possible.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Keep in mind that it's the last 20 bytes of the .pack, but the last 21-40 bytes of the .idx file.
9567e73
to
da9a22e
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Posting comments/questions that I have so far. Still looking through the rest of gvfs-helper.c
builtin/checkout.c
Outdated
new_branch_info->commit, | ||
opts->show_progress); | ||
exit(1); | ||
#endif |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will this code be removed before merging (I'm assuming it's for local testing)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is still a WIP. not sure if i need it or not going forward.
list-objects-filter.c
Outdated
struct object *obj, | ||
const char *pathname, | ||
const char *filename, | ||
void *filter_data_) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Coding in Git question, is there any special significance to this parameter name ending in _
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's no official significance to it.
I just did it to try to keep track of the void* callback context handle
and give it a little more meaning than "data" or "cb" or "ctx" as is
usual.
The callback API wants a void , but the first thing I want to do inside
the function is to cast it to a specific type. So there's always a dance
in these types of functions to assign void to a typed alias. This was
my attempt to make the CB API contract a little more meaningful.
GH__ERROR_CODE__USAGE = -1, /* will be mapped to usage() */ | ||
GH__ERROR_CODE__OK = 0, | ||
GH__ERROR_CODE__ERROR = 1, /* unspecified */ | ||
// GH__ERROR_CODE__CACHE_SERVER_NOT_FOUND = 2, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is this error code commented out?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I overhauled the error return codes and eliminated the need for this one. I intend to delete it.
gvfs-helper.c
Outdated
else if (!strcmp(k, "core.gvfs")) | ||
gh__cmd_opts.is_core_vfs = 1; | ||
|
||
else if (!strcmp(k, "core.scalar")) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Currently Scalar is setting core.gvfs
(with the specific flags it wants) and not core.scalar
is that going to cause any issues?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No. I was anticipating that we'd get a core.scalar flag as part of the global rename. If that's not the case, I can get rid of this.
91a411b
to
a72e8f8
Compare
Rebased forward to tip of feature branch. This version currently omits the "--filter=sparse:cone" code because of a massive conflict with upstream. (And I'm not convinced I still need it.) |
8f3c319
to
576b8bc
Compare
gvfs-helper-client.c
Outdated
v2_path++; | ||
|
||
prepare_alt_odb(the_repository); | ||
for (odb = the_repository->objects->odb; odb; odb = odb->next) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jeffhostetler I think this API could be done significantly differently and make our lives less error prone. The client should pick an ODB based on the config value (and do a platform path conversion to get into some “canonical” form) and then start the helper with that path in mind. The helper could just return a loose object oid or a pack name. We wouldn’t need this loop on every response. We could navigate directly to the odb pointer we have already.
That would at least help the windows bugs I’m having. The Mac test failure needs some more investigation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like that. Let me take a look at that. That will simplify things considerably.
bc19eff
to
3446dd0
Compare
@jeffhostetler with your latest changes (3446dd0) I ran the tests with this one failure in the parallel case:
It is being output in this spot of
|
I added the following line to
This allowed me to run the functional tests with GIT_TRACE2_EVENT pointing to a log file. The relevant events are here:
Specifically, the sid ending in 4722 output the request for the blob first, but then was the one to fail. Maybe the blob is being written by the second process as we try to read it in the first process? |
@jeffhostetler I figured out the problem: the two processes are writing to the same temp file at the same time. One "wins" its rename operation and the other loses. The fix would be one of the following:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We have functional tests passing. Let's get this in!
Signed-off-by: Jeff Hostetler <[email protected]>
Teach subprocess_start() to use a copy of the passed `cmd` string rather than borrowing the buffer from the caller. Some callers of subprocess_start() pass the value returned from find_hook() which points to a static buffer and therefore is only good until the next call to find_hook(). This could cause problems for the long-running background processes managed by sub-process.c where later calls to subprocess_find_entry() to get an existing process will fail. This could cause more than 1 long-running process to be created. TODO Need to confirm, but if only read_object_hook() uses TODO subprocess_start() in this manner, we could drop this TODO commit when we drop support for read_object_hook(). Signed-off-by: Jeff Hostetler <[email protected]>
Add function to start a subprocess with an argv. Signed-off-by: Jeff Hostetler <[email protected]>
Create helper tool to use the GVFS Protocol REST API to
fetch objects and configuration data from a GVFS cache-server
or a primary Git server.
This is V2. It replaces my earlier version from
#190
This version refactors pretty much everything and eliminates most
of the code duplication between the various GET and POST forms.
It also clarifies and centralizes the error handling.
This version eliminates the modifications to http.[ch] and credential.[ch]
that I needed in V1. Everything is now self-contained in gvfs-helper.c.
Usage is the same as before (and described in the TODO comment
at the top of the source file).