Difference in local and VM results and RealWorld DevOPS experience

[cyrusmsk]

Some contributors faces with this problem:
when they testing some changes in the code locally - it showed some improvement in the performance. However, when they make a PR - together with the help of the repo owner they are finding out that the update make a significant regression.

Any ideas from community how to debug such behavior and provide more meaningful changes to the repo?

This problem could be not only for this particular benchmark, but I can imagine real world case: developer find some improvement in the code, test It locally (even on the same Docker settings). However when devops of the company or CI/CD infrastructure deployed the update into corporate cloud or private Kubernetes cluster - instead of reducing in memory/speed they could see regressions.
One of the possible solutions - are testing on similar to production infrastructure and environment. But are there any other possible ways to eliminate such behavior?

[jinyus]

The testing machine is the most powerful VM you can get on Azure FREE tier (1 Month). Azure F4s v2 - 4vCPU-8GB-Ubuntu 22.04

Here is a single command you can use to install docker and build the image on a fresh VM:

# after you login
sudo apt update &&
    sudo apt install -y apt-transport-https ca-certificates curl software-properties-common &&
    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg &&
    echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list >/dev/null &&
    sudo apt update &&
    apt-cache policy docker-ce &&
    sudo apt install -y docker-ce &&
    git clone https://github.com/jinyus/related_post_gen.git &&
    cd related_post_gen &&
    sudo docker build -t databench .

After the image has been built, you can run the test for any/all languages using:

sudo docker run -it --rm -e GIT_REPO=YOUR_REPO_URL -e BRANCH=main -e TEST_NAME=rust  databench

PS: The image takes ~10mins to build as it installs all the runtimes/compilers for all langauges in the test... if you just want to test a single language, you can edit the Dockerfile and comment out the steps that install the compilers that you aren't interested in.