test-gen-api-docs always fail at the first run after the api docs have change

[gongmax]

What happened:
Currently, the test-gen-api-docs simply generates new api doc from the code and compare it with the old api doc (the agones_crd_api_reference.html file in the workspace), and fails if they do not match. So whenever there is change introduced to api doc, the first run of test-gen-api-docs will always fail because the old doc will definitely not match.

What you expected to happen:
We need to rethink about what should be tested against for a "site-test"/"doc-test". What's the failure scenario we want to catch? The current test doesn't provide much value since it can only catch error of accidental api doc change in the first test run, the following run will succeed anyway.

How to reproduce it (as minimally and precisely as possible):
Add a new field to the GameServerAllocationStatus (pkg/apis/allocation/v1/gameserverallocation.go), and run make test.
Anything else we need to know?:

Environment:

• Agones version: dev
• Kubernetes version (use kubectl version): N/A
• Cloud provider or hardware configuration: N/A
• Install method (yaml/helm): N/A
• Troubleshooting guide log(s): N/A
• Others: N/A

[markmandel]

The test is to ensure that a user runs make gen-api-docs along with their changes to a CRD/Extension -- which is the test we care about. It's probably more about CI than about local testing.

[gongmax]

Oh, that makes sense. Then maybe better documentation about local development process would help. I run make test after make changes locally so found this issue a little annoying.

[markmandel]

make test would take a long time I expect! 😄 I usually try and be a bit targeted with which tests I run, and lean on CI for the rest.

[gongmax]

close this for now since it's as expected.