Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scheduler data race #1758

Closed
wants to merge 3 commits into from
Closed

Scheduler data race #1758

wants to merge 3 commits into from

Conversation

crazyhappygame
Copy link
Contributor

@crazyhappygame crazyhappygame commented Mar 17, 2024

Describe your changes

This is next attempt for
#1753

Issue ticket number and link

#1752

Checklist before requesting a review

For each PR

  • Add Copyright if it missed:
    - "Copyright (c) 2019-present Axmol Engine contributors (see AUTHORS.md)."

  • I have performed a self-review of my code.

    Optional:

    • I have checked readme and add important infos to this PR.
    • I have added/adapted some tests too.

For core/new feature PR

  • I have checked readme and add important infos to this PR.
  • I have added thorough tests.

Comments:
Fix for data race:

WARNING: ThreadSanitizer: data race (pid=8716)
  Read of size 8 at 0x000114b03a18 by main thread:
    #0 std::__1::vector<std::__1::function<void ()>, std::__1::allocator<std::__1::function<void ()>>>::empty[abi:v160006]() const vector:552 (cpp-tests:arm64+0x101ec4e24)
    #1 ax::Scheduler::update(float) Scheduler.cpp:855 (cpp-tests:arm64+0x102014d94)
    #2 ax::Director::drawScene() Director.cpp:289 (cpp-tests:arm64+0x101f527e8)
    #3 ax::Director::mainLoop() Director.cpp:1547 (cpp-tests:arm64+0x101f59da0)
    #4 ax::Application::run() Application-mac.mm:76 (cpp-tests:arm64+0x10236560c)
    #5 main main.cpp:32 (cpp-tests:arm64+0x1019a6ce4)

  Previous write of size 8 at 0x000114b03a18 by thread T18 (mutexes: write M0):
    #0 std::__1::enable_if<is_move_constructible<std::__1::function<void ()>*>::value && is_move_assignable<std::__1::function<void ()>*>::value, void>::type std::__1::swap[abi:v160006]<std::__1::function<void ()>*>(std::__1::function<void ()>*&, std::__1::function<void ()>*&) swap.h:40 (cpp-tests:arm64+0x101e96780)
    #1 std::__1::vector<std::__1::function<void ()>, std::__1::allocator<std::__1::function<void ()>>>::__swap_out_circular_buffer(std::__1::__split_buffer<std::__1::function<void ()>, std::__1::allocator<std::__1::function<void ()>>&>&) vector:920 (cpp-tests:arm64+0x101e95ce0)
    #2 void std::__1::vector<std::__1::function<void ()>, std::__1::allocator<std::__1::function<void ()>>>::__emplace_back_slow_path<std::__1::function<void ()>>(std::__1::function<void ()>&&) vector:1584 (cpp-tests:arm64+0x102017524)
    #3 std::__1::function<void ()>& std::__1::vector<std::__1::function<void ()>, std::__1::allocator<std::__1::function<void ()>>>::emplace_back<std::__1::function<void ()>>(std::__1::function<void ()>&&) vector:1603 (cpp-tests:arm64+0x10201433c)
    #4 ax::Scheduler::runOnAxmolThread(std::__1::function<void ()>) Scheduler.cpp:722 (cpp-tests:arm64+0x10201423c)
    #5 ax::AsyncTaskPool::ThreadTasks::ThreadTasks()::'lambda'()::operator()() const AsyncTaskPool.h:137 (cpp-tests:arm64+0x101ee8d9c)
    #6 decltype(std::declval<ax::AsyncTaskPool::ThreadTasks::ThreadTasks()::'lambda'()>()()) std::__1::__invoke[abi:v160006]<ax::AsyncTaskPool::ThreadTasks::ThreadTasks()::'lambda'()>(ax::AsyncTaskPool::ThreadTasks::ThreadTasks()::'lambda'()&&) invoke.h:394 (cpp-tests:arm64+0x101ee8b48)
    #7 void std::__1::__thread_execute[abi:v160006]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, ax::AsyncTaskPool::ThreadTasks::ThreadTasks()::'lambda'()>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, ax::AsyncTaskPool::ThreadTasks::ThreadTasks()::'lambda'()>&, std::__1::__tuple_indices<>) thread:288 (cpp-tests:arm64+0x101ee8af4)
    #8 void* std::__1::__thread_proxy[abi:v160006]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, ax::AsyncTaskPool::ThreadTasks::ThreadTasks()::'lambda'()>>(void*) thread:299 (cpp-tests:arm64+0x101ee8588)

  As if synchronized via sleep:
    #0 nanosleep <null>:264457860 (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x2f618)
    #1 std::__1::this_thread::sleep_for(std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>> const&) <null>:264457860 (libc++.1.dylib:arm64e+0x22b88)
    #2 main main.cpp:32 (cpp-tests:arm64+0x1019a6ce4)

  Location is heap block of size 368 at 0x000114b03900 allocated by main thread:
    #0 operator new(unsigned long) <null>:264457860 (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x84420)
    #1 ax::Director::init() Director.cpp:120 (cpp-tests:arm64+0x101f50280)
    #2 ax::Director::getInstance() Director.cpp:95 (cpp-tests:arm64+0x101f5010c)
    #3 ax::Configuration::loadConfigFile(std::__1::basic_string_view<char, std::__1::char_traits<char>>) Configuration.cpp:401 (cpp-tests:arm64+0x101ef5a58)
    #4 AppDelegate::applicationDidFinishLaunching() AppDelegate.cpp:64 (cpp-tests:arm64+0x1001bb8f0)
    #5 ax::Application::run() Application-mac.mm:57 (cpp-tests:arm64+0x10236554c)
    #6 main main.cpp:32 (cpp-tests:arm64+0x1019a6ce4)

  Mutex M0 (0x000114b03a30) created at:
    #0 pthread_mutex_lock <null>:264457860 (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x3cf8c)
    #1 std::__1::mutex::lock() <null>:264457860 (libc++.1.dylib:arm64e+0x16710)
    #2 std::__1::lock_guard<std::__1::mutex>::lock_guard[abi:v160006](std::__1::mutex&) __mutex_base:94 (cpp-tests:arm64+0x101e934b8)
    #3 ax::Scheduler::runOnAxmolThread(std::__1::function<void ()>) Scheduler.cpp:721 (cpp-tests:arm64+0x102014228)
    #4 ax::AudioEngineImpl::_play2d(ax::AudioCache*, int) AudioEngineImpl.cpp:595 (cpp-tests:arm64+0x101ec29b4)
    #5 decltype(*std::declval<ax::AudioEngineImpl*&>().*std::declval<void (ax::AudioEngineImpl::*&)(ax::AudioCache*, int)>()(std::declval<ax::AudioCache*&>(), std::declval<int&>())) std::__1::__invoke[abi:v160006]<void (ax::AudioEngineImpl::*&)(ax::AudioCache*, int), ax::AudioEngineImpl*&, ax::AudioCache*&, int&, void>(void (ax::AudioEngineImpl::*&)(ax::AudioCache*, int), ax::AudioEngineImpl*&, ax::AudioCache*&, int&) invoke.h:359 (cpp-tests:arm64+0x101edf75c)
    #6 std::__1::__bind_return<void (ax::AudioEngineImpl::*)(ax::AudioCache*, int), std::__1::tuple<ax::AudioEngineImpl*, ax::AudioCache*, int>, std::__1::tuple<>, __is_valid_bind_return<void (ax::AudioEngineImpl::*)(ax::AudioCache*, int), std::__1::tuple<ax::AudioEngineImpl*, ax::AudioCache*, int>, std::__1::tuple<>>::value>::type std::__1::__apply_functor[abi:v160006]<void (ax::AudioEngineImpl::*)(ax::AudioCache*, int), std::__1::tuple<ax::AudioEngineImpl*, ax::AudioCache*, int>, 0ul, 1ul, 2ul, std::__1::tuple<>>(void (ax::AudioEngineImpl::*&)(ax::AudioCache*, int), std::__1::tuple<ax::AudioEngineImpl*, ax::AudioCache*, int>&, std::__1::__tuple_indices<0ul, 1ul, 2ul>, std::__1::tuple<>&&) bind.h:263 (cpp-tests:arm64+0x101edf62c)
    #7 std::__1::__bind_return<void (ax::AudioEngineImpl::*)(ax::AudioCache*, int), std::__1::tuple<ax::AudioEngineImpl*, ax::AudioCache*, int>, std::__1::tuple<>, __is_valid_bind_return<void (ax::AudioEngineImpl::*)(ax::AudioCache*, int), std::__1::tuple<ax::AudioEngineImpl*, ax::AudioCache*, int>, std::__1::tuple<>>::value>::type std::__1::__bind<void (ax::AudioEngineImpl::*)(ax::AudioCache*, int), ax::AudioEngineImpl*, ax::AudioCache*&, int&>::operator()[abi:v160006]<>() bind.h:295 (cpp-tests:arm64+0x101edf574)
    #8 decltype(std::declval<std::__1::__bind<void (ax::AudioEngineImpl::*)(ax::AudioCache*, int), ax::AudioEngineImpl*, ax::AudioCache*&, int&>&>()()) std::__1::__invoke[abi:v160006]<std::__1::__bind<void (ax::AudioEngineImpl::*)(ax::AudioCache*, int), ax::AudioEngineImpl*, ax::AudioCache*&, int&>&>(std::__1::__bind<void (ax::AudioEngineImpl::*)(ax::AudioCache*, int), ax::AudioEngineImpl*, ax::AudioCache*&, int&>&) invoke.h:394 (cpp-tests:arm64+0x101edf514)
    #9 void std::__1::__invoke_void_return_wrapper<void, true>::__call<std::__1::__bind<void (ax::AudioEngineImpl::*)(ax::AudioCache*, int), ax::AudioEngineImpl*, ax::AudioCache*&, int&>&>(std::__1::__bind<void (ax::AudioEngineImpl::*)(ax::AudioCache*, int), ax::AudioEngineImpl*, ax::AudioCache*&, int&>&) invoke.h:487 (cpp-tests:arm64+0x101edf478)
    #10 std::__1::__function::__alloc_func<std::__1::__bind<void (ax::AudioEngineImpl::*)(ax::AudioCache*, int), ax::AudioEngineImpl*, ax::AudioCache*&, int&>, std::__1::allocator<std::__1::__bind<void (ax::AudioEngineImpl::*)(ax::AudioCache*, int), ax::AudioEngineImpl*, ax::AudioCache*&, int&>>, void ()>::operator()[abi:v160006]() function.h:185 (cpp-tests:arm64+0x101edf424)
    #11 std::__1::__function::__func<std::__1::__bind<void (ax::AudioEngineImpl::*)(ax::AudioCache*, int), ax::AudioEngineImpl*, ax::AudioCache*&, int&>, std::__1::allocator<std::__1::__bind<void (ax::AudioEngineImpl::*)(ax::AudioCache*, int), ax::AudioEngineImpl*, ax::AudioCache*&, int&>>, void ()>::operator()() function.h:356 (cpp-tests:arm64+0x101edd6d4)
    #12 std::__1::__function::__value_func<void ()>::operator()[abi:v160006]() const function.h:510 (cpp-tests:arm64+0x1019d556c)
    #13 std::__1::function<void ()>::operator()() const function.h:1156 (cpp-tests:arm64+0x1019d3674)
    #14 ax::AudioCache::invokingPlayCallbacks() AudioCache.cpp:372 (cpp-tests:arm64+0x101e930d8)
    #15 ax::AudioCache::readDataTask(unsigned int) AudioCache.cpp:336 (cpp-tests:arm64+0x101e92d9c)
    #16 ax::AudioEngineImpl::preload(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void (bool)>)::$_1::operator()() const AudioEngineImpl.cpp:516 (cpp-tests:arm64+0x101ed9e68)
    #17 decltype(std::declval<ax::AudioEngineImpl::preload(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void (bool)>)::$_1&>()()) std::__1::__invoke[abi:v160006]<ax::AudioEngineImpl::preload(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void (bool)>)::$_1&>(ax::AudioEngineImpl::preload(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void (bool)>)::$_1&) invoke.h:394 (cpp-tests:arm64+0x101ed9d78)
    #18 void std::__1::__invoke_void_return_wrapper<void, true>::__call<ax::AudioEngineImpl::preload(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void (bool)>)::$_1&>(ax::AudioEngineImpl::preload(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void (bool)>)::$_1&) invoke.h:487 (cpp-tests:arm64+0x101ed9cdc)
    #19 std::__1::__function::__alloc_func<ax::AudioEngineImpl::preload(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void (bool)>)::$_1, std::__1::allocator<ax::AudioEngineImpl::preload(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void (bool)>)::$_1>, void ()>::operator()[abi:v160006]() function.h:185 (cpp-tests:arm64+0x101ed9c88)
    #20 std::__1::__function::__func<ax::AudioEngineImpl::preload(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void (bool)>)::$_1, std::__1::allocator<ax::AudioEngineImpl::preload(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void (bool)>)::$_1>, void ()>::operator()() function.h:356 (cpp-tests:arm64+0x101ed7b80)
    #21 std::__1::__function::__value_func<void ()>::operator()[abi:v160006]() const function.h:510 (cpp-tests:arm64+0x1019d556c)
    #22 std::__1::function<void ()>::operator()() const function.h:1156 (cpp-tests:arm64+0x1019d3674)
    #23 ax::AudioEngine::AudioEngineThreadPool::threadFunc() AudioEngine.cpp:126 (cpp-tests:arm64+0x101ea971c)
    #24 decltype(*std::declval<ax::AudioEngine::AudioEngineThreadPool*&>().*std::declval<void (ax::AudioEngine::AudioEngineThreadPool::*&)()>()()) std::__1::__invoke[abi:v160006]<void (ax::AudioEngine::AudioEngineThreadPool::*&)(), ax::AudioEngine::AudioEngineThreadPool*&, void>(void (ax::AudioEngine::AudioEngineThreadPool::*&)(), ax::AudioEngine::AudioEngineThreadPool*&) invoke.h:359 (cpp-tests:arm64+0x101eadb04)
    #25 std::__1::__bind_return<void (ax::AudioEngine::AudioEngineThreadPool::*)(), std::__1::tuple<ax::AudioEngine::AudioEngineThreadPool*>, std::__1::tuple<>, __is_valid_bind_return<void (ax::AudioEngine::AudioEngineThreadPool::*)(), std::__1::tuple<ax::AudioEngine::AudioEngineThreadPool*>, std::__1::tuple<>>::value>::type std::__1::__apply_functor[abi:v160006]<void (ax::AudioEngine::AudioEngineThreadPool::*)(), std::__1::tuple<ax::AudioEngine::AudioEngineThreadPool*>, 0ul, std::__1::tuple<>>(void (ax::AudioEngine::AudioEngineThreadPool::*&)(), std::__1::tuple<ax::AudioEngine::AudioEngineThreadPool*>&, std::__1::__tuple_indices<0ul>, std::__1::tuple<>&&) bind.h:263 (cpp-tests:arm64+0x101eada24)
    #26 std::__1::__bind_return<void (ax::AudioEngine::AudioEngineThreadPool::*)(), std::__1::tuple<ax::AudioEngine::AudioEngineThreadPool*>, std::__1::tuple<>, __is_valid_bind_return<void (ax::AudioEngine::AudioEngineThreadPool::*)(), std::__1::tuple<ax::AudioEngine::AudioEngineThreadPool*>, std::__1::tuple<>>::value>::type std::__1::__bind<void (ax::AudioEngine::AudioEngineThreadPool::*)(), ax::AudioEngine::AudioEngineThreadPool*>::operator()[abi:v160006]<>() bind.h:295 (cpp-tests:arm64+0x101ead99c)
    #27 decltype(std::declval<std::__1::__bind<void (ax::AudioEngine::AudioEngineThreadPool::*)(), ax::AudioEngine::AudioEngineThreadPool*>>()()) std::__1::__invoke[abi:v160006]<std::__1::__bind<void (ax::AudioEngine::AudioEngineThreadPool::*)(), ax::AudioEngine::AudioEngineThreadPool*>>(std::__1::__bind<void (ax::AudioEngine::AudioEngineThreadPool::*)(), ax::AudioEngine::AudioEngineThreadPool*>&&) invoke.h:394 (cpp-tests:arm64+0x101ead8f0)
    #28 void std::__1::__thread_execute[abi:v160006]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, std::__1::__bind<void (ax::AudioEngine::AudioEngineThreadPool::*)(), ax::AudioEngine::AudioEngineThreadPool*>>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, std::__1::__bind<void (ax::AudioEngine::AudioEngineThreadPool::*)(), ax::AudioEngine::AudioEngineThreadPool*>>&, std::__1::__tuple_indices<>) thread:288 (cpp-tests:arm64+0x101ead89c)
    #29 void* std::__1::__thread_proxy[abi:v160006]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, std::__1::__bind<void (ax::AudioEngine::AudioEngineThreadPool::*)(), ax::AudioEngine::AudioEngineThreadPool*>>>(void*) thread:299 (cpp-tests:arm64+0x101ead330)

  Thread T18 (tid=2140243, running) created by main thread at:
    #0 pthread_create <null>:264457860 (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x3062c)
    #1 std::__1::__libcpp_thread_create[abi:v160006](_opaque_pthread_t**, void* (*)(void*), void*) __threading_support:378 (cpp-tests:arm64+0x100418f88)
    #2 std::__1::thread::thread<ax::AsyncTaskPool::ThreadTasks::ThreadTasks()::'lambda'(), void>(ax::AsyncTaskPool::ThreadTasks::ThreadTasks()::'lambda'()&&) thread:315 (cpp-tests:arm64+0x101ee836c)
    #3 std::__1::thread::thread<ax::AsyncTaskPool::ThreadTasks::ThreadTasks()::'lambda'(), void>(ax::AsyncTaskPool::ThreadTasks::ThreadTasks()::'lambda'()&&) thread:307 (cpp-tests:arm64+0x101ee7d38)
    #4 ax::AsyncTaskPool::ThreadTasks::ThreadTasks() AsyncTaskPool.h:120 (cpp-tests:arm64+0x101ee7c2c)
    #5 ax::AsyncTaskPool::ThreadTasks::ThreadTasks() AsyncTaskPool.h:119 (cpp-tests:arm64+0x101ee7a9c)
    #6 ax::AsyncTaskPool::AsyncTaskPool() AsyncTaskPool.cpp:48 (cpp-tests:arm64+0x101ee79bc)
    #7 ax::AsyncTaskPool::AsyncTaskPool() AsyncTaskPool.cpp:48 (cpp-tests:arm64+0x101ee7874)
    #8 ax::AsyncTaskPool::getInstance() AsyncTaskPool.cpp:37 (cpp-tests:arm64+0x101ee7794)
    #9 void ax::FileUtils::performOperationOffthread<ax::FileUtils::isFileExist(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void (bool)>) const::$_6, std::__1::function<void (bool)>>(ax::FileUtils::isFileExist(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void (bool)>) const::$_6&&, std::__1::function<void (bool)>&&) FileUtils.h:902 (cpp-tests:arm64+0x10226e054)
    #10 ax::FileUtils::isFileExist(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void (bool)>) const FileUtils.cpp:919 (cpp-tests:arm64+0x10226df64)
    #11 TestIsFileExistAsync::onEnter() FileUtilsTest.cpp:916 (cpp-tests:arm64+0x1004cb898)
    #12 ax::Director::setNextScene() Director.cpp:1158 (cpp-tests:arm64+0x101f53170)
    #13 ax::Director::drawScene() Director.cpp:302 (cpp-tests:arm64+0x101f528bc)
    #14 ax::Director::mainLoop() Director.cpp:1547 (cpp-tests:arm64+0x101f59da0)
    #15 ax::Application::run() Application-mac.mm:76 (cpp-tests:arm64+0x10236560c)
    #16 main main.cpp:32 (cpp-tests:arm64+0x1019a6ce4)

SUMMARY: ThreadSanitizer: data race vector:552 in std::__1::vector<std::__1::function<void ()>, std::__1::allocator<std::__1::function<void ()>>>::empty[abi:v160006]() const
==================

{
_performMutex.lock();
// fixed #4123: Save the callback functions, they must be invoked after '_performMutex.unlock()', otherwise if
// new functions are added in callback, it will cause thread deadlock.
auto temp = std::move(_actionsToPerform);
_performMutex.unlock();
_actionsToPerformEmpty.store(false, std::memory_order_release);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this be set to true?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you fixed

@crazyhappygame crazyhappygame requested a review from rh101 March 17, 2024 17:44
@halx99
Copy link
Collaborator

halx99 commented Mar 18, 2024

The schedule actionsToPerform delay depends on timing add by writer thread, even through you use atomic(will add cpu lock when compiled as asm).

@crazyhappygame
Copy link
Contributor Author

The schedule actionsToPerform delay depends on timing add by writer thread, even through you use atomic(will add cpu lock when compiled as asm).

I am so sorry I do not fully understand your comment.
atomic is use here to fix data race and to avoid locking for every iteration (PR #1753)

@smilediver
Copy link
Contributor

smilediver commented Mar 22, 2024

I think you're introducing another race.

        _performMutex.unlock();
       // <--- another thread can add new actions here and set _actionsToPerformEmpty to false here.
        _actionsToPerformEmpty.store(true, std::memory_order_release);

should be

        _actionsToPerformEmpty.store(true, std::memory_order_release);
        _performMutex.unlock();

I would get rid of this atomic and just take the lock and check for emptiness inside. It's called once per frame, so it has close to 0 performance impact, and I wouldn't trade readability here for potential issues (like above).

@crazyhappygame
Copy link
Contributor Author

@smilediver you are right :) I will change it.

the fix with lock was here #1753
Please join discussion:)

@crazyhappygame crazyhappygame closed this by deleting the head repository Apr 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants