GD-205: Nested testing suits

[batomow]

Now that im following unit testing best practices and aiming for higher code coverage i end up with lots of tests within the same suite, which makes it a little hard to read. I noticed that you can collapse the tests by their testing suite, I was wondering if there is a way to nest or group related suites.
So that the ui allows to go from this:

> my suite
  - test a1
  - test a2
  - test a3
  - test b1
  - test b2

To this:

> my suite
  > a test suite
    - test a1
    - test a2
    - test a3
  > b test suite
    - test b1
    - test b2

[MikeSchulze]

Hi @batomow, if I understood you correctly, you have organized your tests incorrectly.

A Test-Suite is a collection of test related to a single source.
You don't should mix tests inside a Test-Suite for different sources.

Best practice is to use the context menu on the source file and let GdUnit generate the Test-Suite and tests.

Test-Suites should organized like:
../src/source_a.gd -> ../test/source_a_test.gd (including only test to cover code inside 'source_a.gd')
../src/source_b.gd -> ../test/source_b_test.gd (including only test to cover code inside 'source_b.gd')

Please send me your feedback, maybe i understood you wrongly.

[batomow]

Hmm A and B aren't different sources. Let me give you a concrete example.
Im testing a class called hashmap.gd. Within this class I have a couple functions, like so:

##HashMap
func rehash_objects(): 
  ..code here
  
func get_neighbors():
  ..code here
  

So my testing suite looks like this

var hashmap:HashMap
func before():
   var hashmap = HashMap.new(48)
   
func before_test():
   hashmap.clear()
   
func rehash_objects_some_test_1():
   ..setup code for rehash
   ...assert_something here
   
func rehash_objects_some_test_2():
   ..setup code for rehash
   ...assert_something here
   
func rehash_objects_some_test_3():
   ..setup code for rehash
   ...assert_something here
   
func get_neighbors_some_test_1():
   ...setup code for neighbors
   ...assert_something here
   
func get_neighbors_some_test_2():
   ...setup code for neighbors
   ...assert_something here
   

Which outputs tests like this.

> my suit
   - rehash_objects_some_test_1 //passed
   - rehash_objects_some_test_2 //passed
   - rehash_objects_some_test_3 //passed
   - get_neighbors_some_test_1  //passed
   - get_neighbors_some_test_1  //passed

The problem here is that I repeated a lot of code, I could use the before_test but the setup code for the first function could potentially interfere with the setup code of the second function and/or make the testing slower. Another work around is to make another testing suite or to make some setup functions that I would call before each test. But all of this is cumbersome.

Now I usually do web development and I've used other testing frameworks, like jest, and this framework allows for grouping and nesting of the tests. Like so

##My test suite
describe ("testing the first function of my class",  ()=>{

     after_all(()=>{...cleanup the code})
     
     before_each(()=>{...settup code})
     
    describe ("this  group will test the first function", ()=>{
        it("tests first function for whatever", ()=>{
            assert_something()
        })
        it("tests first function for something else", ()=>{
            assert_something()
        })
        it("tests first function for something else again", ()=>{
            assert_something()
        })
    })
    
     describe ("this  group will test the second function", ()=>{
        it("tests second function for whatever", ()=>{
            assert_something()
        })
        it("tests first function for something else", ()=>{
            assert_something()
        })
        describe ("this will further test something that requires a particular settup", ()=>{
            before_all(()=>{..more settup code})
            
            it("test some corner case", ()=>{
                assert_something()
            })
            
            it("test some corner case from another angle", ()=>{
                assert_something()
            })
            
        })
    })
    
})

The above example would output like this: 
> my suit
    > this group will test the first function
       - test the first function for whatever
       - test the first function for something else
       - test the first function for something else again
    > this gorup will tst the second function 
       - test the second function for whatever
       - test the second function for something else
       > this will further test something that requires a particular settup
          - test some corner case
          - test some corner case from another angle. 

So I thought it could be nice to have something like this in here.

[MikeSchulze]

GdUnit3 is not designed to group tests at the moment.
You can solve code duplication by using extra functions to setup your test environment.

var _hashmap :HashMap
func before():
   _hashmap = HashMap.new(48)
   
func before_test():
   _hashmap .clear()
  
# helper to setup for rehash testing
func setup_rehash(<args>) -> HashMap:
 .. do setup stuff here
  return _hashmap

# helper to setup for neighborstesting
func setup_neighbors(<args>) -> HashMap:
 .. do setup stuff here
  return _hashmap
 
# tests ...
func test_rehash_objects_some_1():
   setup_rehash(_hashmap )
   ...assert_something here
   
func test_rehash_objects_some_2():
   var map := setup_rehash()
   ...assert_something here
   
func test_rehash_objects_some_3():
    var map := setup_rehash()
   ...assert_something here
   
func test_get_neighbors_some_1():
    var map := setup_neighbors()
   ...assert_something here
   
func test_get_neighbors_some_2():
   var map := setup_neighbors()
   ...assert_something here

If you have a specific set of test parameters, you can use a custom fuzzer to simplify the test code.

var _hashMap :HashMap

func before():
	_hashMap = HashMap.new()

class RehashFuzzer extends Fuzzer:
	# key vs expected hash code
	var _data := {
		10 : 193396939, 
		20: 155869737,
		30 : 2238420049 }
	
	func next_value() -> int:
		return _data.keys()[_iteration_index-1]
		
	func epected_hash_code() -> int:
		return _data.values()[_iteration_index-1] 


func test_rehash(fuzzer = RehashFuzzer.new(), fuzzer_iterations = 3):
	var key = fuzzer.next_value()
	
	_hashMap.add(key, 10)
	_hashMap.rehash_objects()
	assert_int(_hashMap.hash_code()).is_equal(fuzzer.epected_hash_code())

[MikeSchulze]

@batomow, is it ok to close this as clarified?

BTW i have seen your comment about script parsing on godotengine/godot#33624 (comment) and if you interested you can improve the GdUnit3 script parser.
Otherwise if it's ok i want to use your code and implement into GdUnit3?

[batomow]

Sure I am interested but in the meantime let me give you an updated version just in case you need it.

Edit* Is very late at night here and my brain dont work propoerly haha, what i meant was: Yes I can help. Let me just update my comment on the other thread with the latest code.

Yes close it as clarified, thanks!

BTW. I tried implementing the following code to test my code structure and i got the error saying _iteration_index is not defined.

var method_definitions = [
	TU.fun("_init", TYPE_NIL, [
		TU.arg("cellsize", TYPE_INT)
	]),
	TU.fun("clear", TYPE_NIL),
	TU.fun("get_hash", TYPE_DICTIONARY, [
		TU.arg("position", TYPE_VECTOR2)
	]),
	TU.fun("add_object",TYPE_NIL, [
		TU.arg("object", TYPE_NIL)
	]), 
]

class MethodDefinitionFuzzer extends Fuzzers: 
	var data: Array

	func _init(definitions:Array).():
		assert(definitions)
		data = definitions

	func next_value()->Dictionary: 
		assert(not data.empty())
		return data[(_iteration_index-1)%data.size()]
		

[MikeSchulze]

Sounds good ;)
You have an typo in your code. You have to extend from Fuzzer and not from Fuzzers ;)

[MikeSchulze]

close as clarified