Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(tasks): Fix @repeat_every blocking app startup #308

Closed
wants to merge 3 commits into from

Conversation

Keiishu
Copy link
Contributor

@Keiishu Keiishu commented May 23, 2024

Description of the bug

A function using @repeat_every blocks app startup on version 0.6.0 when it is called in the context manager.
This should fix #305.

Description of the changes

Replacing (https://github.com/dmontagu/fastapi-utils/blob/master/fastapi_utils/tasks.py#L77) with asyncio.ensure_future(loop()) works.
Every test passes for me.

@ollz272
Copy link
Contributor

ollz272 commented May 23, 2024

@dmontagu @yuval9313 please can this be looked at asap

@Keiishu
Copy link
Contributor Author

Keiishu commented May 23, 2024

I'm not sure why those tests don't pass, I did run pytest locally and I got no issues whatsoever. Should I have tried with make test instead?
I'll investigate.

@Keiishu Keiishu marked this pull request as draft May 23, 2024 19:41
@Keiishu Keiishu marked this pull request as ready for review May 23, 2024 20:22
@Keiishu
Copy link
Contributor Author

Keiishu commented May 23, 2024

My bad, I don't know what happened with my tests then, but they didn't pass locally neither in the end. They do now ; it was just a missing await.

EDIT: I'm sorry, we're back at the beginning. It blocks FastAPI's context manager.
Looking at Starlette's code (https://github.com/encode/starlette/blob/9f16bf5c25e126200701f6e04330864f4a91a898/starlette/routing.py#L732), it seems the issue stems in the fact that Starlette is waiting for the end of the lifespan, while here, by awaiting the loop, we are never getting out of it. It needs to be a coroutine, but then the tests fail...

@Keiishu Keiishu marked this pull request as draft May 23, 2024 20:46
@@ -74,7 +74,7 @@ async def loop() -> None:
repetitions += 1
await asyncio.sleep(seconds)

await loop()
await asyncio.ensure_future(loop())

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Keiishu : I think you need to remove the await. This is what was used in v0.2.1. I test your code but remove the await in my local FastAPI project, and it worked. It no longer block the app to start.

Copy link
Contributor Author

@Keiishu Keiishu May 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ohandyya : That's what I did in the first commit on this PR, and it works for me, I've used it like this with no issues whatsoever in a relatively big FastAPI project. But then, tests were not passing anymore and I thought it was my fault. So I must say I'm a little bit confused.
Maybe the tests are faulty?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not familiar with sarlette but I'm pretty confident that asyncio.ensure_future doesn't return coroutine, therefore await isn't needed. I'm using the below code since the 0.6.0 repeat_every break, and this works fine for me.

def repeat(seconds: float, delay: float = None):
    def decorator(func):
        is_coroutine = asyncio.iscoroutinefunction(func)
        @wraps(func)
        async def wrapper(*args, **kwargs):
            async def repeat_function():
                if delay:
                    await asyncio.sleep(delay)

                while True:
                    if is_coroutine:
                        await func(*args, **kwargs)
                    else:
                        run_in_threadpool(func, *args, **kwargs)
                    await asyncio.sleep(seconds)

            asyncio.ensure_future(repeat_function())

        return wrapper
    return decorator

Copy link
Contributor Author

@Keiishu Keiishu May 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mgkid3310 You're right, await shouldn't be necessary. But without it, the tests introduced with 0.6.0 fail. And while your code works perfectly fine for me as well, as do the first commit of this PR, the tests all fail.
So yeah, maybe there is an issue with the tests themselves...

I'll revert my second commit (the await), keep this PR as a draft and wait for further insight on what's happening with the tests.

@Keiishu
Copy link
Contributor Author

Keiishu commented May 28, 2024

For now, I'll revert my second commit (the await), keep this PR as a draft and wait for further insight on what's happening with the tests.

This reverts commit 3d9ba0d.

See the conversation on fastapiutils#308 for further details.
@mgkid3310
Copy link
Contributor

mgkid3310 commented Jun 2, 2024

I've reviewed the test_tasks.py code and I guess there's a problem with how repeat_every() is handled in pytest. For example, while test for repeat counter fails as the counter stays at 0 but it expects 3 (max_repetitions) meaning that the iterations didn't work as intended, when I increase the sleep seconds to 5 (that makes total execution time 15 seconds) the test still completes in 0.11s.

So my guess is: the loop() function and the func() (which is the TestRepeatEveryBase.increase_counter()) is not executed at all before the assert self.counter == max_repetitions evaluation. It is true that await increase_counter_task() is valid statement, but since loop() is ensured in the future, we need a different way for assert self.counter == max_repetitions to wait loop() to complete the func() calls.

@mgkid3310
Copy link
Contributor

I'm working on adding a on_complete parameter to repeat_every() and using that to trigger an event once the loop is complete, not sure if it will work but for now I guess it's the only option I see.

@mgkid3310
Copy link
Contributor

#310 I've merged your commits and added my further fixes for tests. I've solved problems for most tests (4 out of 6 failing tests) but failed to solve the remaining 2 before going to bed. Can you take a look at the code?

@yuval9313
Copy link
Collaborator

yuval9313 commented Jun 3, 2024

Hey, this is still marked with a draft, would you like this PR to be reviewed or the other one?

EDIT:
I prefer the #310 so I'm going to focus on merging it

@yuval9313 yuval9313 marked this pull request as ready for review June 3, 2024 21:12
@Keiishu
Copy link
Contributor Author

Keiishu commented Jun 3, 2024

No problem @yuval9313 ; #310 is a fork of this PR with updated tests, so we should focus there. I'll close this one.

@Keiishu
Copy link
Contributor Author

Keiishu commented Jun 3, 2024

#310 I've merged your commits and added my further fixes for tests. I've solved problems for most tests (4 out of 6 failing tests) but failed to solve the remaining 2 before going to bed. Can you take a look at the code?

Nice, thanks! I really can't focus on this right now, until Thursday at least, so I'll take a look at it then if nobody fixed it by then.

@Keiishu Keiishu closed this Jun 3, 2024
yuval9313 added a commit that referenced this pull request Jun 10, 2024
* fix(tasks): Fix @repeat_every blocking app startup

* fix(tasks): Add missing await on ensure_future

* Revert "fix(tasks): Add missing await on ensure_future"

This reverts commit 3d9ba0d.

See the conversation on #308 for further details.

* added on_complete parameter for pytest purpose

* added loop completed state and loop_completed function

* formatting improvements (for mypy)

* use Union for typing support in older Python versions

* on_exception feature added and deprecation warnings for logger and raise_exception added

* Fix TestRepeatEveryWithAsynchronousFunction so that it runs with async functions

* add pytest-timeout to package and fix typo

* fix function type problems by removing conditional types

---------

Co-authored-by: Martin <[email protected]>
Co-authored-by: Yuval Levi <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] @repeat_every blocks app startup on version 0.6.0
5 participants