Refactor propfind tests and add a new one

[fschade]

Description

Today we discussed that we need another propfind test with a deeper nested folder structure, in this breath i refactored those to use a shared lib which makes developing propfind tests more easy. Cc.: @dragotin & @ScharfViktor

To keep the old results meaningful the test with a nesting count of 5 is still there.

Changes

• linted project files
• refactor to shared propfind lib
• add new propfind test with a depth of 45

ToDos

• find a better test naming to keep the files meaningful

[ScharfViktor]

run k6 test with docker ocis and local k6: ./scripts/cdperf --cloud-vendor=ocis --k6-test-host=https://localhost:9200 --k6-docker=false
branch: propfind-tests
env: macOc Big Sur v.11.4 M1

tests were successful:

• deep_1000_files_5_nested_folders.ts
• deep_1000_files_45_nested_folders.ts
• deep_1000_files.ts

small finding, here it try to run file lib.ts:

[fschade]

run k6 test with docker ocis and local k6: ./scripts/cdperf --cloud-vendor=ocis --k6-test-host=https://localhost:9200 --k6-docker=false branch: propfind-tests env: macOc Big Sur v.11.4 M1

tests were successful:

• deep_1000_files_5_nested_folders.ts
• deep_1000_files_45_nested_folders.ts
• deep_1000_files.ts

small finding, here it try to run file lib.ts:

good catch, thanks :) Fixed

[ScharfViktor]

test deep_1000_files_45_nested_folders.ts was very long, maybe the depth is too deep:

deep-1000-files-5-nested-folders.js was running (0h02m14.1s)
propfind-flat-1000-files.js was running (0h00m36.0s)

the result of a remote Intel machine will be faster

[ScharfViktor]

One more thing, the upload test partially failed. This does not seem to be related to your commit, but for your information. Running it again gave the same results

[fschade]

test deep_1000_files_45_nested_folders.ts was very long, maybe the depth is too deep:

deep-1000-files-5-nested-folders.js was running (0h02m14.1s) propfind-flat-1000-files.js was running (0h00m36.0s)

the result of a remote Intel machine will be faster

cc: @dragotin we need to come up with an concept hof those tests, maybe we can test with a smaller amount of files and still can get a meaningful result... ideas? I say we need to go to the lab and brainstorm

[fschade]

One more thing, the upload test partially failed. This does not seem to be related to your commit, but for your information. Running it again gave the same results

we need to check and investigate, if the flakyness is in the tests.... this is not good. I bet it's the backend because we hammer it too much, then its a good thing and cdpers presents us the weak spots

[dragotin]

Agreed. If there are failing requests in the the tests, we need to recognize that immediately and fix the backend. The full success of the test should be an acceptance criteria for releases etc.

I think the amount of files should not be the problem. It should be a for the test reasonable, and kind of "serious" amount of files...

[fschade]

Agreed. If there are failing requests in the the tests, we need to recognize that immediately and fix the backend. The full success of the test should be an acceptance criteria for releases etc.

I think the amount of files should not be the problem. It should be a for the test reasonable, and kind of "serious" amount of files...

i mentioned the amount because it takes 30 minutes now to get an answer, maybe we can also reduce the amount of files inside a nested folder, be faster and be able to answer the same questions.... just thinking loudly ;)

[fschade]

@dragotin can we discuss the new tests this week, dont wanna have this opened for too long

[individual-it]

I've ran those tests from 49.12.163.166 on 49.12.163.186 and they take too long

also some fail

[dragotin]

@individual-it @fschade and me had a brief discussion yesterday and he proposed to run this test with less files to make it run faster. WIP I guess ;-)

[dragotin]

@fschade @ScharfViktor it would be cool to get this in :-)

[dragotin]

My approval, but keep in mind: I am a javascript noob.

[dragotin]

We added a test description doc in docs/tests.md.

Do you mind rebasing and adding some description about these new tests? Thanks a lot.

[fschade]

@individual-it i understand that the tests took really long… but I think this is not a test that will run too often, it’s more or less something to really get some pressure to the instance. I will refactor the test later and try to get it best of both worlds.

[individual-it]

@fschade what response would you expect if the server is under load? can we code that into the test?
And then the question is if we want to run it also nightly, if yes it would be confusing if there are some failures that we tolerate, because s.o. needs to remember what is tolarable

[fschade]

@individual-it in my opinion its ok if the test fails because of high load. We should not reduce the load to make the result green, instead we should finetune the server to be able to handle it.

[individual-it]

@fschade yes I totally agree, I didn't meant to reduce the load, but wanted to ask what should happen under high load and if the server reacts correctly now.
E.g. if under high load the server replies with 500 Internal Server Error I would consider that a bug, if it returns 429 Too Many Requests it should be OK similar if its a slow 20x

I assumed the server reacts under high load already as it should, if that is the case then I would love to get the tests green.
If the server currently does not what it should do under load, then its a bug and the tests should be red (and hopefully the bug fixed at some point). Still then we need some automated system to tell us what are acceptable failures and what not. Not to cover up any problems but not to get into the habit of: "oh we know some tests fail, we can ignore those". Because potentially those failures will grow over time and not followed up.
E.g. we could setup the graphana notifications to shout only starting above a certain (problematic, but acceptable) threshold.

[dragotin]

Let me try to summarize:

• It is not ok if the system fails with 500 under high load, but it (probably) is if it returns an 492.
• Currently we can not really detect that in our framework. That would be a todo.
• To keep the tests meaningful, they have to be green

As long as we can not check if 500 = red test or 492 = green I agree, we need to tweak the test that does not run into the overload. OTOH it would make sense to have this test case to provoke the 500 situation and debug it.

Long story short: For now, we absolutely need a non failing version of the test. Either by debugging the server, enhancing the test to distinguish reply codes or lower the load on the server.

I vote for the latter for now.

[fschade]

thanks for all your feedback, i created a less heavy version of the test. Should be good to go

[ScharfViktor]

just one small find

updated the tests to be less pressing and time consuming

[individual-it]

this resulted in a funny graph https://grafana.k6.infra.owncloud.works/d/8IV_hHt7k/test-durations?orgId=1&from=1644487427677&to=1644755529309

seems like multiple results of the test run are reported. Any idea what happened there?