Invalidate test cache when fixtures change

[matejchalk]

Each test target - unit-test, int-test, e2e-test - should be invalidated whenever the tested code changes, or the tests themselves change. Currently, the cache isn't being invalidated when fixtures change - i.e. files which aren't imported in code, but are read from file system with node:fs module (usually copied to a temporary test folder in beforeEach/beforeAll).

Because of this, we run the risk of merging broken tests, which will be hard to track down.

Steps to reproduce

1. Run npx nx int-test plugin-eslint. Tests will pass and Nx will save cache.
2. Change any file under packages/plugin-eslint/mocks/fixtures in a way that should break some test (e.g. delete nx-monorepo/nx.json).
3. Re-run npx nx int-test plugin-eslint. Tests will pass instantly, Nx logs that cache was used.
4. Run npx nx int-test plugin-eslint --skipNxCache. Observe that tests fail.

This is repeatable for any other type of test that relies on file system - i.e. some integration tests, most E2E tests.

Proposed solution

See discussion, mainly this comment

Acceptance criteria

• Any change to files under {projectRoot} should invalidate test targets in Nx cache.

[BioPhoton]

Can config with the utils project.. the mocks folder is not included in the derived inputs.

[BioPhoton]

I found the error!

We used mocks in .nxignore.

A solution would be to use our helpers from test-nx-utils to create the test files

[matejchalk]

Huh, didn't realize that was there. Seems like there was a good reason, though - if I remove it and try to run tests, then I get this error:

 NX   Failed to process project graph.

8 errors occurred while processing files for the @push-based/nx-verdaccio plugin
.
  - e2e/ci-e2e/mocks/fixtures/npm-workspaces/package.json: No project name found in configuration at e2e/ci-e2e/mocks/fixtures/npm-workspaces. Please ensure the project has a name defined in project.json or package.json.
  - e2e/plugin-js-packages-e2e/mocks/fixtures/npm-repo/package.json: No project name found in configuration at e2e/plugin-js-packages-e2e/mocks/fixtures/npm-repo. Please ensure the project has a name defined in project.json or package.json.
  - packages/ci/mocks/fixtures/monorepos/npm/package.json: No project name found in configuration at packages/ci/mocks/fixtures/monorepos/npm. Please ensure the project has a name defined in project.json or package.json.
  - packages/ci/mocks/fixtures/monorepos/nx/package.json: No project name found in configuration at packages/ci/mocks/fixtures/monorepos/nx. Please ensure the project has a name defined in project.json or package.json.
  - packages/ci/mocks/fixtures/monorepos/pnpm/package.json: No project name found in configuration at packages/ci/mocks/fixtures/monorepos/pnpm. Please ensure the project has a name defined in project.json or package.json.
  - packages/ci/mocks/fixtures/monorepos/turbo/package.json: No project name found in configuration at packages/ci/mocks/fixtures/monorepos/turbo. Please ensure the project has a name defined in project.json or package.json.
  - packages/ci/mocks/fixtures/monorepos/yarn/package.json: No project name found in configuration at packages/ci/mocks/fixtures/monorepos/yarn. Please ensure the project has a name defined in project.json or package.json.
  - packages/plugin-eslint/mocks/fixtures/nx-monorepo/package.json: No project name found in configuration at packages/plugin-eslint/mocks/fixtures/nx-monorepo. Please ensure the project has a name defined in project.json or package.json.
The following projects are defined in multiple locations:
- cli: 
  - packages/ci/mocks/fixtures/monorepos/npm/packages/cli
  - packages/ci/mocks/fixtures/monorepos/nx/packages/cli
  - packages/cli
  - packages/plugin-eslint/mocks/fixtures/nx-monorepo/packages/cli
- core: 
  - packages/ci/mocks/fixtures/monorepos/npm/packages/core
  - packages/ci/mocks/fixtures/monorepos/nx/packages/core
  - packages/core
  - packages/plugin-eslint/mocks/fixtures/nx-monorepo/packages/core
- utils: 
  - packages/ci/mocks/fixtures/monorepos/npm/packages/utils
  - packages/ci/mocks/fixtures/monorepos/nx/packages/utils
  - packages/plugin-eslint/mocks/fixtures/nx-monorepo/packages/utils
  - packages/utils
- nx-plugin: 
  - packages/nx-plugin
  - packages/plugin-eslint/mocks/fixtures/nx-monorepo/packages/nx-plugin

To fix this, set a unique name for each project in a project.json inside the project's root. If the project does not currently have a project.json, you can create one that contains only a name.

Seems like Nx is trying to include our test monorepos 😬

[BioPhoton]

A solution would be to use our helpers from test-nx-utils to create Nx repositories at runtime under tmp

[BioPhoton]

For now to be more fine-grained we could list all paths separately and later refactor one by one.

[matejchalk]

A solution would be to use our helpers from test-nx-utils to create Nx repositories at runtime under tmp

I guess, but I'd much prefer to maintain these fixtures using files and folders.

What about if we just rename the files which have special meaning to Nx? In mocks/fixtures, we'd replace project.json and nx.json with something like _project.json and _nx.json. We'd copy the folder to tmp as usual before running tests, but we'd add an extra step which renames _project.jsons and _nx.jsons to project.json and nx.json (we can implement a test utility function do this recursively). As long as tmp is ignored by Nx, it shouldn't interfere with normal Nx projects.

[BioPhoton]

Then lets delete the other helper that creates the monorepo setup and only use one way.

[hanna-skryl]

I tried to implement the suggested _nx.json/_project.json renaming approach, but ran into a couple of problems:

1. It broke the ESLint plugin's Nx integration tests. Despite the test setup copying fixtures to a tmp directory and renaming the files back, the test isolation wasn't working properly, and it was still seeing all 28 projects from the main workspace.
2. More importantly, the cache still wasn't getting invalidated when fixture files changed.

What actually worked was using targeted .nxignore patterns:

**/mocks/**/package.json
**/mocks/**/nx.json
**/mocks/**/project.json

Instead of blocking the whole mocks directory, this just blocks specific files that make Nx think the fixtures are real projects. Is this targeted ignore approach acceptable as the solution?

[matejchalk]

Is this targeted ignore approach acceptable as the solution?

If it works, then sure 👍 Although I'm guessing changes to {nx,project,package}.json won't register as cache invalidations? That's still better everything in mocks being ignored.

[hanna-skryl]

Although I'm guessing changes to {nx,project,package}.json won't register as cache invalidations?

Right, changes in these files won't invalidate the cache.

[BioPhoton]

What about adding **/tmp/** to .nxignore?