Add support for tests with coroutines

[dave08]

Would it be possible to add cdescribe, ccontext, cBeforeEachTest and cit, etc.. (for example), that would have a runBlocking (or the new TestCoroutine system...) inside... that would avoid extra nesting (like mockk.io did...).

[arturdryomov]

The general idea at this point is to delegate these things to tests. Otherwise it would expand to RxJava, Future, ListenableFuture and so on. Spek is a testing engine, we don’t even provide asserts. You can go wild and create your own extensions though.

Side note — from a personal point of view, such things should be a part of code structure and test arrangement. I’m not into coroutines but with RxJava it is possible to provide and replace Scheduler using IoC of choice. It has little to do with the testing framework.

[dave08]

I just thought that since coroutines is really part of Kotlin core, and Spek is a Kotlin test framework, it would just avoid lots of boilerplate code... but I guess one could make extension functions...

[arturdryomov]

Future is a part of the JVM and yet JUnit does not have direct helpers to deal with them. The same goes to Thread and Executor. Working with them is part of the codebase, as well as anything related to multithreading or anything else.

It is not a boilerplate, it is dealing with application level-abstractions. Nothing can deal with that good enough except the developer. Test engine has a single task — run tests. And it is not trivial already.

[artem-zinnatullin]

I stand with @ming13 here, it should be possible to provide Coroutines support as extension project on top of Spek, similarly to how we provide Specification style right now.

By providing custom style you'll have control on what is allowed and how it works if there are tweaks required for Coroutines.

We might however reconsider this and provide it as a Spek's own module in future :)

Closing for now.

[oripwk]

Hi,

What's the status on this? Will this be available in Spek 2?

[artem-zinnatullin]

I'd still recommend to build coroutines support as separate extension library.

In the conversation above you can find a lot of good points made by @ming13 on why we should probably not add such functionality into the core.

Btw, it's not just coroutines, we've cut on most extensions in 2.x to release sooner and spend less time on support.

[jcornaz]

I'd still recommend to build coroutines support as separate extension library.

Is there a chance that spekframework provide that extension library?

Please also note, that suspend is a language keyword. We are not talking about supporting any arbitrary library. We only ask about support of the Kotlin language.

[ed-george]

Having Spek without coroutines support, or any roadmap to supporting coroutines is a serious limitation. It would be great to see this implemented asap!

[raniejade]

As I don't use coroutines that much, I'd like to see the various pain points you guys encounter while testing coroutines using Spek. Honestly, the biggest issue here is I'm not entirely sure how this extension would look like.

[jcornaz]

@raniejade In order to avoid boilerplate in each specification we have to add an equivalent of the following in each project:

fun <R> LifeCycleAware.memoizedBlocking(mode: CachingMode = defaultCachingMode, factory: suspend () -> R, destructor: suspend (R) -> Unit): MemoizedValue<R> =
	memoized(mode, factory = { runBlocking { factory() } }, destructor = { runBlocking { destructor(it) } })
	
fun LifeCycleAware.beforeBlocking(block: suspend () -> Unit) =
	beforeGroup { runBlocking { block() } }
	
fun LifeCycleAware.beforeEachBlocking(block: suspend () -> Unit) =
	beforeEachTest { runBlocking { block() } }
	
fun LifeCycleAware.afterBlocking(block: suspend () -> Unit) =
	afterGroup { runBlocking { block() } }
	
fun LifeCycleAware.afterEachBlocking(block: suspend () -> Unit) =
	afterEachTest { runBlocking { block() } }

Note that:

1. My naming scheme is debatable.
2. I don't create extension for describe because its content is only test-discovery and not testing. So there is never need to call suspending function in describe.
3. I don't create extension for it because I only put assertions there. I never need to call suspending function from a it.
4. I have no clue about how to make it work for Kotlin/JS where runBlocking is not possible because of the single-threaded nature of Javascript. I know that almost all javascript test engines support asynchronous testing, but I don't have enough experience to help there.

[ed-george]

That looks like a great start @jcornaz. But can this issue be re-opened or re-created?

[raniejade]

thanks @jcornaz! Re-opening (so it's easier to track) but I'm not committing to anything yet.

[raniejade]

Crazy idea: So you can call normal functions in a suspend block but not vice versa and to actually suspend is explicit (via the builders), why not add the suspend modifier to blocks/lambdas that participate in the execution phase (fixtures, memoized and test)? This means pulling in kotlinx-coroutines-core library but I will probably pull it in as part of implementing test timeouts (#643).

[jcornaz]

why not add the suspend modifier to blocks/lambdas that participate in the execution phase (fixtures, memoized and test)

Yes indeed. It'd be great from a usage perspective. I just wonder if it would noticeably impact performances or not.

This means pulling in kotlinx-coroutines-core library

Yes, but it can be an implementation detail of Spek (use implementationin gradle to make the dependency not transitive). The only API is the keyword suspend which is part of the kotlin the language. Technically you could provide suspend lambdas and the desired functionnality without using kotlinx.coroutines at all. But of course I wouldn't advise to do that, just re-use the runBlocking provided by kotlinx-coroutines-core

[raniejade]

Interesting, KotlinTest did the same approach: kotest/kotest#332.

I just wonder if it would noticeably impact performances or not.

Probably not that significant? AFAIK suspend just desugars to adding an extra parameter. The actual suspending logic (don't quote me on this) is done via a library and probably only happens when you call any of the builders. So if you don't anything async there shouldn't be any significant impact on the runtime vs being called in a normal block.

[raniejade]

I'll do some more thinking... if you have any more ideas keep them coming.

[raniejade]

I have timeouts now working (#643 - at least for JVM, K/N only supports a single thread coroutine) which is using coroutines under the hood. Would be really easy now to just add in the suspend keywords all over the place. Though I'm still not convinced if this is the correct thing to do.

[jcornaz]

Though I'm still not convinced if this is the correct thing to do.

I don't know. I'd say it sound somehow appealing. As far as I could see, there is a tiny performance impact on the first test only (probably because more classes have to be loaded). But I could not notice clear impact for the rest of the tests.

An other "cons" is that adding suspend break binary compatibility. I'm not sure how important it is for a testing framework. But better to know it.

But otherwise, I have to say it is really tempting to say: "Yes, put suspend all over the place". It would be so great to not have to worry if we call suspending function or not in spek tests.

[raniejade]

I started experimenting with this on #835.

[sondr3]

Drive-by question, is this working in the latest version of Spek or am I missing someting? I still have to wrap my tests with runBlocking.