Stuck in middle of a scan

[frobnitzem]

I was happily scanning along, and ran into an error cloning a repository. The code won't say which, but based on the last success, it was probably https://github.com/laristra/ristra.

Anyway, now there are two issues. How can I fix the code, and how can I resume the scan without re-doing 5 hours of (serial) work?

[2:10:37<17:22,  3.32s/it]✅ Routine (ANL-CESAR/XSBench --> CommitInfoMiningRoutineRequest) was successful!
 18%|█████████████▎                                                           | 70/383  18%|████████████▊                                                         | 70/383 [2:10:41<9:44:23, 112.02s/it]
Traceback (most recent call last):
  File "/Users/99r/work/ideas_ecp/repo-env/bin/reposcanner", line 33, in <module>
    sys.exit(load_entry_point('reposcanner', 'console_scripts', 'reposcanner')())
  File "/Users/99r/work/ideas_ecp/reposcanner/src/reposcanner/reposcanner.py", line 135, in run
    scannerMain(args)
  File "/Users/99r/work/ideas_ecp/reposcanner/src/reposcanner/reposcanner.py", line 69, in scannerMain
    manager.run(repositoriesDataFile,credentialsDataFile,configDataFile)
  File "/Users/99r/work/ideas_ecp/reposcanner/src/reposcanner/manager.py", line 284, in run
    self.executeWithNoGUI()
  File "/Users/99r/work/ideas_ecp/reposcanner/src/reposcanner/manager.py", line 293, in executeWithNoGUI
    task.process(self._routines+self._analyses,self._store,self._notebook)
  File "/Users/99r/work/ideas_ecp/reposcanner/src/reposcanner/manager.py", line 61, in process
    self._response = selectedAgent.run(self._request)
  File "/Users/99r/work/ideas_ecp/reposcanner/src/reposcanner/routines.py", line 46, in run
    response = self.execute(request)
  File "/Users/99r/work/ideas_ecp/reposcanner/src/reposcanner/routines.py", line 93, in execute
    raise e
  File "/Users/99r/work/ideas_ecp/reposcanner/src/reposcanner/routines.py", line 83, in execute
    session = pygit2.clone_repository(request.getRepositoryLocation().getURL(),
  File "/Users/99r/work/ideas_ecp/repo-env/lib/python3.9/site-packages/pygit2/__init__.py", line 222, in clone_repository
    payload.check_error(err)
  File "/Users/99r/work/ideas_ecp/repo-env/lib/python3.9/site-packages/pygit2/callbacks.py", line 93, in check_error
    check_error(error_code)
  File "/Users/99r/work/ideas_ecp/repo-env/lib/python3.9/site-packages/pygit2/errors.py", line 65, in check_error
    raise GitError(message)
_pygit2.GitError: remote authentication required but no callback set

[elaineraybourn]

Hey David, I know the project lead of Ristra if for some reason that is helpful to you.

[rmmilewi]

@frobnitzem I'll look into this issue! The error message

_pygit2.GitError: remote authentication required but no callback set

is very informative. Probably just need to add in a time-out or other such callback to make sure Reposcanner doesn't get stuck when trying to clone an inaccessible repository.

The other part of this is to make a standard feature for all data mining routines to skip over doing mining routines that already have generated files associated with them (something we can enable via a CLI argument). I added this in for CommitInfoMiningRoutine on lines 39-42, but it really should be a standard feature.