cifuzz: get_replace_repo_and_build_command can overwrite artifacts copied from Dockerfile

[DavidKorczynski]

Integrating CIFuzz into DuckDB led to an issue where a fuzzer located in OSS-Fuzz gets copied into $SRC/duckdb/fuzznamer.cpp but is then deleted. I believe it's related to rm -rf {rm_path} here

oss-fuzz/infra/cifuzz/continuous_integration.py

Line 156 in a75844d

command = (f'cd / && rm -rf {rm_path} && cp -r {host_repo_path} '

The logs are as follows:

2021-11-03T12:47:32.9399816Z Step 5/5 : COPY parse_fuzz_test.cpp $SRC/duckdb/parse_fuzz_test.cpp
...
...
2021-11-03T12:47:44.9956838Z 2021-11-03 12:47:44,995 - root - INFO - Running: docker run --rm --privileged --cap-add SYS_PTRACE -e FUZZING_ENGINE=libfuzzer -e ARCHITECTURE=x86_64 -e CIFUZZ=True -e SANITIZER=address -e FUZZING_LANGUAGE=c++ -e OUT=/github/workspace/build-out --volumes-from 03cfbe88ed21 gcr.io/oss-fuzz/duckdb /bin/bash -c 'cd / && rm -rf /src/duckdb/* && cp -r /github/workspace/storage/duckdb /src && cd - && compile'.
2021-11-03T12:47:47.6753094Z /src/duckdb

As such, later the build process would fail as the fuzzer is no longer there.

Am not sure if this is by design or not, but putting up the issue as I can imagine a fair number of projects may use the Dockerfile to put stuff into their repository folders.

[DavidKorczynski]

This is the DuckDB PR: duckdb/duckdb#2501

[jonathanmetzman]

This is a known issue. I can try to look into this next week.
CC @oliverchang

[oliverchang]

The OSV integration copies out the repository from the image instead of checking out its own (which would include any extra non-tracked files that are copied into it). Perhaps we can just do the same thing here?

[DavidKorczynski]

Any updates on this one?

[jonathanmetzman]

Starting on this now.

[jonathanmetzman]

The OSV integration copies out the repository from the image instead of checking out its own (which would include any extra non-tracked files that are copied into it). Perhaps we can just do the same thing here?

I'm starting to implement this. But i think it suffers from a flaw.
Suppose the change we are testing is an update to a submodule.
It's not enough to checkout this change, we also need to update the submodules.
Does OSV account for this? Can you point me to the code that does this?

The alternative would be making users clone their repo with github's clone action which is annoying because:

1. It's a breaking API change.
2. github's clone makes diffing really hard. I haven't figured out how to make them play nice.

[oliverchang]

It's not enough to checkout this change, we also need to update the submodules.
Does OSV account for this? Can you point me to the code that does this?

I don't think we handle submodules today (I thought we did). We could in theory always call checkout with --recurse-submodules or something similar.