hashFiles of a non-wildcard reads the whole tree

[michaelfig]

Refs Agoric/agoric-sdk#502
Refs actions/cache#177

Describe the bug
I expect hashFiles('yarn.lock') only to read $workspace/yarn.lock.

Instead, it tries to read the entire working directory subtree. That's unacceptable when a build may create millions of files.

To Reproduce
Steps to reproduce the behavior:

1. Have a cache like:

    - name: cache node modules
      uses: actions/cache@v1
      with:
        path: ~/.cache/yarn
        key: ${{ runner.os }}-yarn-${{ hashFiles('yarn.lock') }}
        restore-keys: |
          ${{ runner.os }}-yarn-

2. Create millions of files during a later step
3. Run the action at least twice to prime the cache
4. See that in the post job cleanup all those directory entries were scanned, causing the step to waste time (in my case, minutes)

Expected behavior

No time-consuming traversal of the entire directory tree just to read one file. That single hash takes 00:01:45 in a big tree when it should be milliseconds.

Runner Version and Platform

Version of your runner? Github.com current

OS of the machine running the runner? Ubuntu

What's not working?

Cache plugin cleanup takes ages even though there are no changes (cache hit).

Job Log Output

From https://github.com/Agoric/agoric-sdk/actions/runs/34932285

Wed, 05 Feb 2020 06:17:00 GMT
##[debug]....Evaluating runner:
Wed, 05 Feb 2020 06:17:00 GMT
##[debug]....=> Object
Wed, 05 Feb 2020 06:17:00 GMT
##[debug]....Evaluating String:
Wed, 05 Feb 2020 06:17:00 GMT
##[debug]....=> 'os'
Wed, 05 Feb 2020 06:17:00 GMT
##[debug]..=> 'Linux'
Wed, 05 Feb 2020 06:17:00 GMT
##[debug]..Evaluating hashFiles:
Wed, 05 Feb 2020 06:17:00 GMT
##[debug]....Evaluating String:
Wed, 05 Feb 2020 06:17:00 GMT
##[debug]....=> 'yarn.lock'
Wed, 05 Feb 2020 06:17:00 GMT
##[debug]Search root directory: '/home/runner/work/agoric-sdk/agoric-sdk'
Wed, 05 Feb 2020 06:17:18 GMT
##[debug]Search pattern: '/home/runner/work/agoric-sdk/agoric-sdk/yarn.lock'
Wed, 05 Feb 2020 06:18:16 GMT
##[debug]Found 2525470 files

Note how the above line took almost a full minute to execute.

Here's the rest of the step:

Wed, 05 Feb 2020 06:18:16 GMT
##[debug]1 matches to hash
Wed, 05 Feb 2020 06:18:16 GMT
##[debug]Hash /home/runner/work/agoric-sdk/agoric-sdk/yarn.lock
Wed, 05 Feb 2020 06:18:16 GMT
##[debug]Final hash result: 'c9a52e06293c448028063bd3e12465978bf2a97846ee469acd64cf676678609e'
Wed, 05 Feb 2020 06:18:16 GMT
##[debug]..=> 'c9a52e06293c448028063bd3e12465978bf2a97846ee469acd64cf676678609e'
Wed, 05 Feb 2020 06:18:16 GMT
##[debug]=> 'Linux-yarn-c9a52e06293c448028063bd3e12465978bf2a97846ee469acd64cf676678609e'
  Complete job

Runner and Worker's Diagnostic Logs

N/a

[michaelfig]

I would suggest that the hashFiles implementation could find the shortest common non-wildcard prefix of the supplied patterns (stopping at a segment containing wildcards), and only read the actual filesystem starting at that prefix.

So:

• /home/runner/work/agoric-sdk/agoric-sdk/mydir/** would read the tree starting at /home/runner/work/agoric-sdk/agoric-sdk/mydir
• /home/runner/work/agoric-sdk/agoric-sdk/yarn.lock would read the tree starting at /home/runner/work/agoric-sdk/agoric-sdk/yarn.lock
• /home/runner/work/agoric-sdk/agoric-sdk/**/yarn.lock would read the tree starting at /home/runner/work/agoric-sdk/agoric-sdk
• /home/runner/work/agoric-sdk/agoric-sdk/mydir/**\n/home/runner/work/agoric-sdk/agoric-sdk/yarn.lock would read the tree starting at /home/runner/work/agoric-sdk/agoric-sdk

[michaelfig]

I see that this was fixed in #268

Where would I find out the ETA for Github Actions to be upgraded to this version?

[ericsciple]

@TingluoHuang do you know whats the state of the runner release that contains glob fixes?

[TingluoHuang]

The runner is in ring 0, but we need service change to deploy in order to control the rollout

[TingluoHuang]

this should be fixed now.