Vitest Coverage Differs locally from Github Actions

[jperetzIL]

Hi,

I have been dealing with this issue for a bit and haven't found a solution or anything about it online. I have just moved to vite and vitest from CRA and Jest. While most things were straight forward and worked great - I ran into an issue with testing coverage.

Here is my Vite config File

export default defineConfig(() => {
  return {
    define: {
      'process.env': {},
    },
    resolve: {
      alias: {
        UI: path.resolve(__dirname, './src/UI'),
        features: path.resolve(__dirname, './src/features'),
        hooks: path.resolve(__dirname, './src/hooks'),
        context: path.resolve(__dirname, './src/context'),
        lib: path.resolve(__dirname, './src/lib'),
        store: path.resolve(__dirname, './src/store'),
        api: path.resolve(__dirname, './src/api'),
        'test-utils': path.resolve(__dirname, './src/test-utils'),
      },
    },
    server: {
      open: true,
      port: '3000',
    },
    build: {
      outDir: 'build',
    },
    test: {
      globals: true,
      environment: 'jsdom',
      setupFiles: './tests/setup.js',
      coverage: {
        all: true,
        provider: 'istanbul', // or 'v8',
        reporter: ['text', 'json', 'html'],
        include: ['src/'],
        exclude: [
          'node_modules/',
          'src/stories/',
          'src/store/',
          'api/',
          'src/test-utils',
        ],
        thresholdAutoUpdate: true,
        lines: 29.16,
        functions: 26.03,
        branches: 23.08,
        statements: 28.93,
        cleanOnRerun: true,
      },
    },
    plugins: [react(), eslint(), svgr()],
  };
});

Currently when I run vitest run --coverage locally I receive the following coverage:

But then when I run the same test through a github action:

name: Tests

on:
  pull_request:
    branches:
      - stage

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - name: Install Modules
        run: npm install --legacy-peer-deps
      - name: Run Tests
        run: npm run test:coverage
        env:
          CI: true

I get this:

As you can see - the Branches coverage is off by .1% and causing it to fail. I tried this with v8 as well and it did the same thing, but by .4% instead.

Any thoughts on what I might be missing here?

[AriPerkkio]

To debug this I would start by making sure CI environment matches with local one. Check that you have package-lock.json committed to version control and maybe even use npm ci command to install the dependencies. Also try running with CI=true flag locally and see if issue comes up then. When using coverage.provider: 'v8', the NodeJS version mismatch might affect the results.

I haven't heard of similar issues before. Happy to look at minimal reproduction if you could provide one.

[jperetzIL]

Thanks for reaching out. I am using Instanbul here instead of V8 since V8 doesn't seem to calculate coverage globally the same way. I took your advice and ran it with CI locally and npm ci and now am getting lower numbers locally and higher numbers on github.

At this point I am going to keep trying for bit longer but might move back to Jest. Jest worked with the same tests before perfectly.
I'll keep this discussion open for now just in case I stay on Vitest or figure it out, but will close it if no progress happens.

Thanks for offering to reproduce, but I cannot expose to much info from my codebase. I can share the package.json if it helps, but that may be all

"engines": {
    "node": "18.15.0",
    "npm": "9.5.0"
  },
  "dependencies": {
    "@date-io/date-fns": "1.3.13",
    "@hookstate/core": "4.0.0-rc21",
    "@material-ui/core": "4.12.3",
    "@material-ui/icons": "4.11.3",
    "@material-ui/lab": "4.0.0-alpha.60",
    "@material-ui/pickers": "3.3.10",
    "@reduxjs/toolkit": "1.8.1",
    "@sentry/react": "6.18.1",
    "@sentry/tracing": "6.18.0",
    "@types/google-libphonenumber": "7.4.23",
    "@types/lodash": "4.14.182",
    "@types/omgopass": "3.2.1",
    "@types/react-datepicker": "^4.10.0",
    "antd": "4.19.3",
    "classnames": "2.3.2",
    "date-fns": "2.21.1",
    "google-libphonenumber": "3.2.27",
    "interweave": "13.0.0",
    "js-cookie": "3.0.1",
    "lodash": "4.17.21",
    "moment": "2.29.4",
    "omgopass": "3.2.1",
    "react": "17.0.2",
    "react-date-range": "1.3.0",
    "react-datepicker": "4.10.0",
    "react-dom": "17.0.2",
    "react-phone-input-2": "2.15.0",
    "react-phone-number-input": "3.1.21",
    "react-redux": "7.2.8",
    "react-router-dom": "6.3.0",
    "react-vis": "1.12.1",
    "use-resize-observer": "9.0.2",
    "uuid": "9.0.0",
    "vite-plugin-svgr": "^4.0.0",
    "web-vitals": "2.1.4"
  },
  "scripts": {
    "start": "vite",
    "build": "vite build",
    "serve": "vite preview",
    "test": "vitest",
    "test:coverage": "vitest run --coverage",
    "lint": "eslint . --ext .js,.jsx,.ts,.tsx",
    "lint-strict": "eslint . --ext .js,.jsx,.ts,.tsx --max-warnings=0",
    "storybook": "storybook dev -p 6006",
    "build-storybook": "storybook build"
  },
  "eslintConfig": {
    "extends": [
      "react-app",
      "react-app/jest",
      "plugin:storybook/recommended"
    ],
    "rules": {
      "react-hooks/rules-of-hooks": "error",
      "react-hooks/exhaustive-deps": "warn"
    }
  },
  "browserslist": {
    "production": [
      ">0.2%",
      "not dead",
      "not op_mini all"
    ],
    "development": [
      "last 1 chrome version",
      "last 1 firefox version",
      "last 1 safari version"
    ]
  },
  "devDependencies": {
    "@storybook/addon-essentials": "^7.0.24",
    "@storybook/addon-interactions": "^7.0.24",
    "@storybook/addon-links": "^7.0.24",
    "@storybook/blocks": "^7.0.24",
    "@storybook/preset-create-react-app": "^7.0.24",
    "@storybook/react": "^7.0.24",
    "@storybook/react-webpack5": "^7.0.24",
    "@storybook/testing-library": "^0.0.14-next.2",
    "@testing-library/jest-dom": "5.16.3",
    "@testing-library/react": "12.0.0",
    "@testing-library/user-event": "14.0.4",
    "@types/jest": "27.4.0",
    "@types/js-cookie": "3.0.1",
    "@types/node": "17.0.23",
    "@types/react": "17.0.39",
    "@types/react-date-range": "1.1.6",
    "@types/react-dom": "17.0.14",
    "@types/react-phone-number-input": "3.0.6",
    "@types/react-redux": "7.1.23",
    "@types/react-vis": "1.11.7",
    "@types/uuid": "9.0.0",
    "@vitejs/plugin-react": "^4.0.4",
    "@vitest/coverage-istanbul": "^0.34.5",
    "@vitest/coverage-v8": "^0.34.5",
    "babel-plugin-named-exports-order": "^0.0.2",
    "chokidar": "3.5.2",
    "eslint": "^8.47.0",
    "eslint-config-react-app": "^7.0.1",
    "eslint-plugin-storybook": "^0.6.12",
    "jsdom": "^22.1.0",
    "prettier": "2.6.1",
    "prop-types": "^15.8.1",
    "storybook": "^7.0.24",
    "typescript": "^5.1.6",
    "vite": "^4.4.9",
    "vite-plugin-eslint": "^1.8.1",
    "vitest": "^0.34.2"
  }
}

[AriPerkkio]

Before debugging any further I would recommend to make sure your Vitest packages versions match.

    "@vitest/coverage-istanbul": "^0.34.5",
    "@vitest/coverage-v8": "^0.34.5",
    "vitest": "^0.34.2"

Also making sure that package-lock.json file is identical on CI and local is essential here. If it's not committed to Git remote, debugging environment differences is kind of useless.

[alec-oms]

@jperetzIL did you get to the bottom of this or find any related issues?

[Mankore]

I am using gitlab and we cache stuff for certain jobs, and gitlab downloads it on start of the job.
for me the solution was to exclude cache folder:

exclude: [ '**/{cache}', ...]