Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LibGit2 integration and tracking subdirs of Julia #104

Merged
merged 2 commits into from
Jun 26, 2018
Merged

LibGit2 integration and tracking subdirs of Julia #104

merged 2 commits into from
Jun 26, 2018

Conversation

timholy
Copy link
Owner

@timholy timholy commented Jun 22, 2018

This is a rebased & extended version of #91 (CC @maleadt). The goal is to provide the means to track both Core.Compiler and the stdlibs, thus fixing #63.

UPDATE: the rest of this post is outdated, but it is retained for posterity

However, I'm encountering a strange LibGit2-related error, and I'm wondering if @omus might have any insights. The issue is that I can't replicate an error if I run what appears to be the same code from the REPL.

Here's the error:

julia> Revise.track(Core.Compiler)   # succeeds

julia> using Base64

julia> Revise.track(Base64)
ERROR: GitError(Code:ERROR, Class:OS, the global/xdg file 'attributes' doesn't exist: No such file or directory)
Stacktrace:
 [1] treewalk(::Function, ::LibGit2.GitTree, ::Bool) at /home/tim/src/julia-1.0/usr/share/julia/stdlib/v0.7/LibGit2/src/tree.jl:38
 [2] treewalk at /home/tim/src/julia-1.0/usr/share/julia/stdlib/v0.7/LibGit2/src/tree.jl:22 [inlined]
 [3] getindex(::LibGit2.GitTree, ::String) at /home/tim/src/julia-1.0/usr/share/julia/stdlib/v0.7/LibGit2/src/tree.jl:177
 [4] track(::Module, ::String, ::LibGit2.GitTree, ::String) at /home/tim/.julia/dev/Revise/src/Revise.jl:319
 [5] track_subdir_from_git(::Module, ::String) at /home/tim/.julia/dev/Revise/src/Revise.jl:392
 [6] track(::Module) at /home/tim/.julia/dev/Revise/src/Revise.jl:377
 [7] top-level scope at none:0

That line is here. Now let's try it manually:

julia> subdir = "/home/tim/src/julia-1.0/base/compiler"
"/home/tim/src/julia-1.0/base/compiler"

julia> repo, repo_path = Revise.git_repo(subdir)
(LibGit2.GitRepo("/home/tim/src/julia-1.0"), "/home/tim/src/julia-1.0")

julia> prefix = subdir[length(repo_path)+2:end]
"base/compiler"

julia> tree = Revise.git_tree(repo, Base.GIT_VERSION_INFO.commit)
GitTree:
Owner: LibGit2.GitRepo("/home/tim/src/julia-1.0")
Number of entries: 29


julia> files = Iterators.filter(file->startswith(file, prefix), keys(tree))
Base.Iterators.Filter{getfield(Main, Symbol("##5#6")),Array{String,1}}(getfield(Main, Symbol("##5#6"))(), [".circleci/config.yml", ".freebsdci.sh", ".gitattributes", ".github/CODE_OF_CONDUCT.md", ".github/ISSUE_TEMPLATE.md", ".github/SUPPORT.md", ".gitignore", ".mailmap", ".travis.yml", "CONTRIBUTING.md"    "test/tuple.jl", "test/unicode/utf8.jl", "test/util/segfault.jl", "test/util/throw_error_exception.jl", "test/vecelement.jl", "test/version.jl", "test/worlds.jl", "ui/.gitignore", "ui/Makefile", "ui/repl.c"])

julia> file = first(files)
"base/compiler/abstractinterpretation.jl"

julia> blob = tree[file];

julia> using LibGit2
WARNING: using LibGit2.hex in module Main conflicts with an existing identifier.

julia> src = LibGit2.content(blob);

julia> src[1:100]
"# This file is a part of Julia. License is MIT: https://julialang.org/license\n\n#############\n# const"

I don't understand what would cause it to work at the REPL and fail in code. Some kind of race condition? I added a sleep in a random location but that didn't seem to help.

@timholy timholy changed the title WIP: LibGit2 integration and tracking subdirs of Julia LibGit2 integration and tracking subdirs of Julia Jun 23, 2018
@timholy
Copy link
Owner Author

timholy commented Jun 23, 2018

Think I got it. Added a line

    ccall((:giterr_clear, :libgit2), Cvoid, ())  # necessary to avoid errors like "the global/xdg file 'attributes' doesn't exist: No such file or directory"

before doing tree[file].

@timholy timholy force-pushed the teh/git branch 2 times, most recently from 5c06d0f to 0acd403 Compare June 23, 2018 22:19
@codecov
Copy link

codecov bot commented Jun 23, 2018

Codecov Report

Merging #104 into master will decrease coverage by 6.05%.
The diff coverage is 54.68%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #104      +/-   ##
==========================================
- Coverage   91.23%   85.18%   -6.06%     
==========================================
  Files           6        8       +2     
  Lines         331      378      +47     
==========================================
+ Hits          302      322      +20     
- Misses         29       56      +27
Impacted Files Coverage Δ
src/pkgs.jl 100% <ø> (ø) ⬆️
src/Revise.jl 87.58% <100%> (+0.32%) ⬆️
src/recipes.jl 25.71% <25.71%> (ø)
src/git.jl 88.88% <88.88%> (ø)
src/delete_method.jl 89.18% <0%> (-0.29%) ⬇️
src/parsing.jl 92.45% <0%> (-0.14%) ⬇️
src/relocatable_exprs.jl 97.14% <0%> (-0.08%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 782e332...e220d60. Read the comment docs.

@timholy timholy force-pushed the teh/git branch 5 times, most recently from 8526344 to 9f65139 Compare June 24, 2018 16:01
@timholy
Copy link
Owner Author

timholy commented Jun 24, 2018

OK, this seems to be working. For the stdlibs you need JuliaLang/julia#27759. @maleadt, any interest in testing this? I haven't tried an out-of-tree build, and I think you mentioned you had.

prefix = subdir[length(repo_path)+2:end] # git-relative path of this subdir
tree = git_tree(repo, Base.GIT_VERSION_INFO.commit)
files = Iterators.filter(file->startswith(file, prefix), keys(tree))
ccall((:giterr_clear, :libgit2), Cvoid, ()) # necessary to avoid errors like "the global/xdg file 'attributes' doesn't exist: No such file or directory"
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll look into this more but I think this is pointing to a LibGit2 issue.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Details on the issue have been filed with libgit2. As a fix for the Julia LibGit2 library I've proposed JuliaLang/julia#27763 which should cleanly work around the issue. For now your giterr_clear solution will work just fine.

@maleadt
Copy link
Contributor

maleadt commented Jun 26, 2018

Thanks for taking over! Tracking Core.Compiler works fine, but stdlibs don't:

   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: https://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.7.0-beta.16 (2018-06-26 03:01 UTC)
 _/ |\__'_|_|_|\__'_|  |  Commit d4942573d9* (0 days old master)
|__/                   |  x86_64-pc-linux-gnu

julia> using Revise
[ Info: Recompiling stale cache file /home/tbesard/Julia/depot/compiled/v0.7/Revise/M1Qo.ji for module Revise

julia> Revise.track(InteractiveUtils)
┌ Warning: Error evaluating expression in Core.Compiler:
│ module InteractiveUtils
| ...
│ end
└ @ Revise parsing.jl:163
ERROR: could not open file editless.jl
Stacktrace:
 [1] include at ./boot.jl:317 [inlined]
 [2] include at ./compiler/compiler.jl:19 [inlined]
 [3] include(::String) at ./stdlib/InteractiveUtils/src/InteractiveUtils.jl:5
 [4] top-level scope at none:0
 [5] eval at ./boot.jl:319 [inlined]
 [6] parse_module!(::Dict{Module,Revise.ExprsSigs}, ::Expr, ::Symbol, ::Module) at /home/tbesard/Julia/depot/packages/Revise/RfHN/src/parsing.jl:161
 [7] parse_expr!(::Dict{Module,Revise.ExprsSigs}, ::Expr, ::Symbol, ::Module) at /home/tbesard/Julia/depot/packages/Revise/RfHN/src/parsing.jl:121
 [8] parse_source!(::Dict{Module,Revise.ExprsSigs}, ::String, ::Symbol, ::Int64, ::Module) at /home/tbesard/Julia/depot/packages/Revise/RfHN/src/parsing.jl:70
 [9] track_subdir_from_git(::Module, ::String) at /home/tbesard/Julia/depot/packages/Revise/RfHN/src/recipes.jl:60
 [10] track(::Module) at /home/tbesard/Julia/depot/packages/Revise/RfHN/src/recipes.jl:33
 [11] top-level scope at none:0

The REPL is gloriously broken after this:

julia> 1+1
Revise is currently tracking the following files: ["/home/tbesard/Julia/depot/packages/Revise/RfHN/src/recipes.jl", "/home/tbesard/Julia/depot/packages/Revise/RfHN/src/relocatable_exprs.jl", "/home/tbesard/Julia/depot/packages/Revise/RfHN/src/types.jl", "/home/tbesard/Julia/depot/packages/Revise/RfHN/src/delete_method.jl", "/home/tbesard/Julia/depot/packages/Revise/RfHN/src/git.jl", "/home/tbesard/Julia/depot/packages/Revise/RfHN/src/parsing.jl", "/home/tbesard/Julia/depot/packages/Revise/RfHN/src/pkgs.jl", "/home/tbesard/Julia/depot/packages/Revise/RfHN/src/Revise.jl"]
ERROR: /home/tbesard/Julia/julia-dev/stdlib/InteractiveUtils/src/InteractiveUtils.jl is not currently being tracked.
Stacktrace:
 [1] error(::String, ::String) at ./error.jl:42
 [2] revise_file_now(::String) at /home/tbesard/Julia/depot/packages/Revise/RfHN/src/Revise.jl:227
 [3] revise(::REPL.REPLBackend) at /home/tbesard/Julia/depot/packages/Revise/RfHN/src/Revise.jl:286
 [4] macro expansion at /home/tbesard/Julia/depot/packages/Revise/RfHN/src/Revise.jl:400 [inlined]
 [5] (::getfield(Revise, Symbol("##33#34")){REPL.REPLBackend})() at ./task.jl:257

julia> 1+1
2

julia> 1+2
2

julia> 1+2
3

Doesn't seem like it picked up tracking the stdlib:

julia> isfile("/home/tbesard/Julia/julia-dev/stdlib/InteractiveUtils/src/InteractiveUtils.jl")
true

julia> Revise.module2files
Dict{Symbol,Array{String,1}} with 1 entry:
  :Revise => ["/home/tbesard/Julia/depot/packages/Revise/RfHN/src/Revise.jl", "/home/tbesard/Julia/depot/packages/Revise/RfHN/src/relocatable_exprs.jl", "/home/tbesard/Julia/depot/packages/Revise/RfHN/src/types.jl", "/home/tbesard/Julia…

@timholy
Copy link
Owner Author

timholy commented Jul 27, 2018

Hmm, works just fine for me; I used it heavily to develop several recent Julia PRs. (Note that currently this may segfault-on-exit due to JuliaLang/julia#28306.)

Not quite sure how to go about debugging this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants