Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Storage crash when running tck #2783

Closed
Shylock-Hg opened this issue Sep 2, 2021 · 6 comments · Fixed by #2795
Closed

Storage crash when running tck #2783

Shylock-Hg opened this issue Sep 2, 2021 · 6 comments · Fixed by #2795
Assignees
@critical27
Labels
type/bug Type: something is unexpected
Milestone
v2.6.0

Comments

@Shylock-Hg
Copy link
Contributor

Please check the FAQ documentation before raising an issue

Please check the FAQ documentation and old issues before raising an issue in case someone has asked the same question that you are asking.

Describe the bug (must be provided)

(lldb) bt
* thread #1, name = 'nebula-storaged', stop reason = signal SIGABRT
  * frame #0: 0x00007f38315bf53f libc.so.6`__GI_raise(sig=6) at raise.c:51
    frame #1: 0x00007f38315a9895 libc.so.6`__GI_abort at abort.c:79
    frame #2: 0x0000000003d16892 nebula-storaged`rocksdb::port::PthreadCall(char const*, int) (.part.0) + 50
    frame #3: 0x0000000003d16990 nebula-storaged`rocksdb::port::Mutex::Lock() + 32
    frame #4: 0x0000000003ab8ecd nebula-storaged`rocksdb::(anonymous namespace)::CleanupIteratorState(void*, void*) + 61
    frame #5: 0x0000000003c9c82e nebula-storaged`rocksdb::Cleanable::~Cleanable() + 30
    frame #6: 0x0000000003b3cc2a nebula-storaged`rocksdb::DBIter::~DBIter() + 1210
    frame #7: 0x0000000003d323ad nebula-storaged`rocksdb::ArenaWrappedDBIter::~ArenaWrappedDBIter() + 29
    frame #8: 0x00000000038d063e nebula-storaged`std::default_delete<rocksdb::Iterator>::operator(this=0x00007f3801c00068, __ptr=0x00007f37ed01fa00)(rocksdb::Iterator*) const at unique_ptr.h:81
    frame #9: 0x00000000038d05a4 nebula-storaged`std::unique_ptr<rocksdb::Iterator, std::default_delete<rocksdb::Iterator> >::~unique_ptr(this=0x00007f3801c00068) at unique_ptr.h:284
    frame #10: 0x00000000038d0947 nebula-storaged`nebula::kvstore::RocksPrefixIter::~RocksPrefixIter(this=0x00007f3801c00060) at RocksEngine.h:55
    frame #11: 0x00000000038d097b nebula-storaged`nebula::kvstore::RocksPrefixIter::~RocksPrefixIter(this=0x00007f3801c00060) at RocksEngine.h:55
    frame #12: 0x000000000244051e nebula-storaged`std::default_delete<nebula::kvstore::KVIterator>::operator(this=0x00007f37eddfbbd0, __ptr=0x00007f3801c00060)(nebula::kvstore::KVIterator*) const at unique_ptr.h:81
    frame #13: 0x000000000243e974 nebula-storaged`std::unique_ptr<nebula::kvstore::KVIterator, std::default_delete<nebula::kvstore::KVIterator> >::~unique_ptr(this=0x00007f37eddfbbd0) at unique_ptr.h:284
    frame #14: 0x00000000025b7027 nebula-storaged`nebula::storage::RebuildEdgeIndexTask::buildIndexGlobal(this=0x00007f3820ea0b00, space=114, part=5, items=size=1) at RebuildEdgeIndexTask.cpp:169
    frame #15: 0x000000000259b01e nebula-storaged`nebula::storage::RebuildIndexTask::invoke(this=0x00007f3820ea0b00, space=114, part=5, items=size=1) at RebuildIndexTask.cpp:100
    frame #16: 0x00000000025adb5c nebula-storaged`nebula::cpp2::ErrorCode std::__invoke_impl<nebula::cpp2::ErrorCode, nebula::cpp2::ErrorCode (nebula::storage::RebuildIndexTask::*&)(int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > > const&), nebula::storage::RebuildIndexTask*&, int&, int&, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > >&>((null)=__invoke_memfun_deref @ 0x00007f37eddfc0f0, __f=0x00007f3801c4c6c0, __t=0x00007f3801c4c6f0, __args=0x00007f3801c4c6ec, __args=0x00007f3801c4c6e8, __args=size=1)(int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > > const&), nebula::storage::RebuildIndexTask*&, int&, int&, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > >&) at invoke.h:73
    frame #17: 0x00000000025ad96f nebula-storaged`std::__invoke_result<nebula::cpp2::ErrorCode (nebula::storage::RebuildIndexTask::*&)(int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > > const&), nebula::storage::RebuildIndexTask*&, int&, int&, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > >&>::type std::__invoke<nebula::cpp2::ErrorCode (__fn=0x00007f3801c4c6c0, __args=0x00007f3801c4c6f0, __args=0x00007f3801c4c6ec, __args=0x00007f3801c4c6e8, __args=size=1)(int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > > const&), nebula::storage::RebuildIndexTask*&, int&, int&, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > >&>(nebula::cpp2::ErrorCode (nebula::storage::RebuildIndexTask::*&)(int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > > const&), nebula::storage::RebuildIndexTask*&, int&, int&, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > >&) at invoke.h:95
    frame #18: 0x00000000025ad8e7 nebula-storaged`nebula::cpp2::ErrorCode std::_Bind<nebula::cpp2::ErrorCode (nebula::storage::RebuildIndexTask::* (nebula::storage::RebuildIndexTask*, int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > >))(int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > > const&)>::__call<nebula::cpp2::ErrorCode, 0ul, 1ul, 2ul, 3ul>(this=0x00007f3801c4c6c0, __args=0x00007f37eddfc1d8, (null)=_Index_tuple<0, 1, 2, 3> @ 0x00007f37eddfc1b0) at functional:400
    frame #19: 0x00000000025ad7f5 nebula-storaged`nebula::cpp2::ErrorCode std::_Bind<nebula::cpp2::ErrorCode (nebula::storage::RebuildIndexTask::* (nebula::storage::RebuildIndexTask*, int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > >))(int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > > const&)>::operator(this=0x00007f3801c4c6c0)<nebula::cpp2::ErrorCode>() at functional:482
    frame #20: 0x00000000025ad266 nebula-storaged`std::_Function_handler<nebula::cpp2::ErrorCode (), std::_Bind<nebula::cpp2::ErrorCode (nebula::storage::RebuildIndexTask::* (nebula::storage::RebuildIndexTask*, int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > >))(int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > > const&)> >::_M_invoke(__functor=0x00007f37eddfc4d8) at std_function.h:285
    frame #21: 0x00000000025725ae nebula-storaged`std::function<nebula::cpp2::ErrorCode ()>::operator(this=0x00007f37eddfc4d8)() const at std_function.h:690
    frame #22: 0x000000000257203e nebula-storaged`nebula::storage::AdminSubTask::invoke(this=0x00007f37eddfc4d8) at AdminTask.h:29
    frame #23: 0x00000000025703a8 nebula-storaged`nebula::storage::AdminTaskManager::runSubTask(this=0x00000000051e8b80, handle=(first = 161, second = 0)) at AdminTaskManager.cpp:163
    frame #24: 0x0000000002587224 nebula-storaged`void std::__invoke_impl<void, void (nebula::storage::AdminTaskManager::*&)(std::pair<int, int>), nebula::storage::AdminTaskManager*&, std::pair<int, int>&>((null)=__invoke_memfun_deref @ 0x00007f37eddfc660, __f=0x00007f3801c00620, __t=0x00007f3801c00638, __args=0x00007f3801c00630)(std::pair<int, int>), nebula::storage::AdminTaskManager*&, std::pair<int, int>&) at invoke.h:73
    frame #25: 0x00000000025870fd nebula-storaged`std::__invoke_result<void (nebula::storage::AdminTaskManager::*&)(std::pair<int, int>), nebula::storage::AdminTaskManager*&, std::pair<int, int>&>::type std::__invoke<void (__fn=0x00007f3801c00620, __args=0x00007f3801c00638, __args=0x00007f3801c00630)(std::pair<int, int>), nebula::storage::AdminTaskManager*&, std::pair<int, int>&>(void (nebula::storage::AdminTaskManager::*&)(std::pair<int, int>), nebula::storage::AdminTaskManager*&, std::pair<int, int>&) at invoke.h:95
    frame #26: 0x00000000025870a7 nebula-storaged`void std::_Bind<void (nebula::storage::AdminTaskManager::* (nebula::storage::AdminTaskManager*, std::pair<int, int>))(std::pair<int, int>)>::__call<void, 0ul, 1ul>(this=0x00007f3801c00620, __args=0x00007f37eddfc708, (null)=_Index_tuple<0, 1> @ 0x00007f37eddfc6e0) at functional:400
    frame #27: 0x0000000002587025 nebula-storaged`void std::_Bind<void (nebula::storage::AdminTaskManager::* (nebula::storage::AdminTaskManager*, std::pair<int, int>))(std::pair<int, int>)>::operator(this=0x00007f3801c00620)<void>() at functional:482
    frame #28: 0x0000000002586d90 nebula-storaged`void folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (p=0x00007f3801c0e760))(std::pair<int, int>)> >(folly::detail::function::Data&) at Function.h:385
    frame #29: 0x00000000041850ad nebula-storaged`folly::ThreadPoolExecutor::runTask(std::shared_ptr<folly::ThreadPoolExecutor::Thread> const&, folly::ThreadPoolExecutor::Task&&) + 173
    frame #30: 0x000000000417966f nebula-storaged`___lldb_unnamed_symbol3984$$nebula-storaged + 63
    frame #31: 0x00000000041e98fe nebula-storaged`bool folly::AtomicNotificationQueue<folly::Function<void ()> >::drive<folly::EventBase::FuncRunner&>(folly::EventBase::FuncRunner&) + 158
    frame #32: 0x00000000041eab5d nebula-storaged`non-virtual thunk to folly::EventBaseAtomicNotificationQueue<folly::Function<void ()>, folly::EventBase::FuncRunner>::handlerReady(unsigned short) + 45
    frame #33: 0x00000000042ae593 nebula-storaged`___lldb_unnamed_symbol4286$$nebula-storaged + 1363
    frame #34: 0x00000000042aec67 nebula-storaged`event_base_loop + 1287
    frame #35: 0x00000000041e5a26 nebula-storaged`folly::EventBase::loopBody(int, bool) + 2134
    frame #36: 0x00000000041e5ebf nebula-storaged`folly::EventBase::loop() + 47
    frame #37: 0x00000000041e7fc4 nebula-storaged`folly::EventBase::loopForever() + 20
    frame #38: 0x0000000004179ff9 nebula-storaged`folly::IOThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>) + 793
    frame #39: 0x0000000004186537 nebula-storaged`void folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::* (folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) + 71
    frame #40: 0x000000000237e4c3 nebula-storaged`folly::detail::function::FunctionTraits<void ()>::operator(this=0x00007f3801c311d0)() at Function.h:400
    frame #41: 0x00000000023d978a nebula-storaged`folly::NamedThreadFactory::newThread(this=0x00007f3801c311d0)>&&)::'lambda'()::operator()() at NamedThreadFactory.h:40
    frame #42: 0x00000000023d973b nebula-storaged`void std::__invoke_impl<void, folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::'lambda'()>((null)=__invoke_other @ 0x00007f37eddfcc60, __f=0x00007f3801c311d0)>&&)::'lambda'()&&) at invoke.h:60
    frame #43: 0x00000000023d96eb nebula-storaged`std::__invoke_result<folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::'lambda'()>::type std::__invoke<folly::NamedThreadFactory::newThread(__fn=0x00007f3801c311d0)>&&)::'lambda'()>(folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::'lambda'()&&) at invoke.h:95
    frame #44: 0x00000000023d96c3 nebula-storaged`void std::thread::_Invoker<std::tuple<folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::'lambda'()> >::_M_invoke<0ul>(this=0x00007f3801c311d0, (null)=_Index_tuple<0> @ 0x00007f37eddfcca0) at thread:244
    frame #45: 0x00000000023d9693 nebula-storaged`std::thread::_Invoker<std::tuple<folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::'lambda'()> >::operator(this=0x00007f3801c311d0)() at thread:251
    frame #46: 0x00000000023d9537 nebula-storaged`std::thread::_State_impl<std::thread::_Invoker<std::tuple<folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::'lambda'()> > >::_M_run(this=0x00007f3801c311c0) at thread:195
    frame #47: 0x00000000046c0210 nebula-storaged`execute_native_thread_routine + 16
    frame #48: 0x00007f383175558e libpthread.so.0`start_thread(arg=<unavailable>) at pthread_create.c:486
    frame #49: 0x00007f3831684513 libc.so.6`__GI___clone at clone.S:95

Your Environments (must be provided)

  • OS: uname -a
  • Compliler: g++ --version or clang++ --version
  • CPU: lscpu
  • Commit id (e.g. a3ffc7d8)

How To Reproduce(must be provided)

Steps to reproduce the behavior:

run tck

Expected behavior

A clear and concise description of what you expected to happen.

Additional context

Provide logs and configs, or any other context to trace the problem.
@Shylock-Hg Shylock-Hg added the type/bug Type: something is unexpected label Sep 2, 2021
@critical27
Copy link
Contributor

What is errno of pthread_mutex_lock?

@Shylock-Hg
Copy link
Contributor Author

What is errno of pthread_mutex_lock?

Lost the scene.

@cangfengzhs
Copy link
Contributor

How to reproduce?

@Shylock-Hg
Copy link
Contributor Author

How to reproduce?

Just run tck, but not stable.

@Sophie-Xie Sophie-Xie added this to the v2.6.0 milestone Sep 3, 2021
@jamieliu1023 jamieliu1023 mentioned this issue Sep 4, 2021
Closed
@cangfengzhs
Copy link
Contributor

cangfengzhs commented Sep 5, 2021 •
edited
Loading

I get another different stack.It is abort by LOG(FATAL).But I am not sure if there is any relationship with the above mentioned problem.

#0  0x00007fe8a0d6b387 in raise () from /lib64/libc.so.6
#1  0x00007fe8a0d6ca78 in abort () from /lib64/libc.so.6
#2  0x0000000004430e7d in google::LogMessage::Fail() ()
#3  0x0000000004435c5e in google::LogMessage::SendToLog() ()
#4  0x0000000004430b4e in google::LogMessage::Flush() ()
#5  0x00000000044313a9 in google::LogMessageFatal::~LogMessageFatal() ()
#6  0x0000000002bab0f9 in nebula::kvstore::RateLimiter::consume (this=0x7fe85a849460, spaceId=77, 
    partId=3, toConsume=0) at /home/hs.zhang/repos/nebula/src/kvstore/RateLimiter.h:43
#7  0x0000000002bae1ea in nebula::storage::RebuildIndexTask::writeOperation (this=0x7fe89186a400, 
    space=77, part=3, batchHolder=0x7fe85c20e300)
    at /home/hs.zhang/repos/nebula/src/storage/admin/RebuildIndexTask.cpp:243
#8  0x0000000002bad537 in nebula::storage::RebuildIndexTask::buildIndexOnOperations (
    this=0x7fe89186a400, space=77, part=3)
--Type <RET> for more, q to quit, c to continue without paging--
   Task.cpp:166
#9  0x0000000002bac92c in nebula::storage::RebuildIndexTask::invoke (this=0x7fe89186a400, space=77, 
    part=3, items=...) at /home/hs.zhang/repos/nebula/src/storage/admin/RebuildIndexTask.cpp:109
#10 0x0000000002bba0db in std::__invoke_impl<nebula::cpp2::ErrorCode, nebula::cpp2::ErrorCode (nebula::storage::RebuildIndexTask::*&)(int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > > const&), nebula::storage::RebuildIndexTask*&, int&, int&, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > >&> (__f=
    @0x7fe871226700: (nebula::cpp2::ErrorCode (nebula::storage::RebuildIndexTask::*)(nebula::storage::RebuildIndexTask * const, int, int, const std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > > &)) 0x2bac2fc <nebula::storage::RebuildIndexTask::invoke(int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > > const&)>, __t=@0x7fe871226730: 0x7fe89186a400, 
    __args#0=@0x7fe87122672c: 77, __args#1=@0x7fe871226728: 3, __args#2=...)
    at /opt/vesoft/toolset/gcc/9.1.0/include/c++/9.1.0/bits/invoke.h:73
#11 0x0000000002bb7f3d in std::__invoke<nebula::cpp2::ErrorCode (nebula::storage::RebuildIndexTask::*&)(int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > > const&), nebula::storage::RebuildIndexTask*&, int&, int&, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > >&> (__fn=
    @0x7fe871226700: (nebula::cpp2::ErrorCode (nebula::storage::RebuildIndexTask::*)(nebula::storage::RebuildIndexTask * const, int, int, const std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > > &)) 0x2bac2fc <nebula::storage::RebuildIndexTask::invoke(int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > > const&)>, __args#0=@0x7fe871226730: 0x7fe89186a400, 
    __args#1=@0x7fe87122672c: 77, __args#2=@0x7fe871226728: 3, __args#3=...)
    at /opt/vesoft/toolset/gcc/9.1.0/include/c++/9.1.0/bits/invoke.h:95
#12 0x0000000002bb61d8 in std::_Bind<nebula::cpp2::ErrorCode (nebula::storage::RebuildIndexTask::*(nebula::storage::RebuildIndexTask*, int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > >))(int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > > const&)>::__call<nebula::cpp2::ErrorCode, , 0ul, 1ul, 2ul, 3ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul, 2ul, 3ul>) (this=0x7fe871226700, __args=...)
    at /opt/vesoft/toolset/gcc/9.1.0/include/c++/9.1.0/functional:400
#13 0x0000000002bb3eb8 in std::_Bind<nebula::cpp2::ErrorCode (nebula::storage::RebuildIndexTask::*(nebula::storage::RebuildIndexTask*, int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std--Type <RET> for more, q to quit, c to continue without paging--
::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > >))(int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > > const&)>::operator()<, nebula::cpp2::ErrorCode>() (this=0x7fe871226700)
    at /opt/vesoft/toolset/gcc/9.1.0/include/c++/9.1.0/functional:484
#14 0x0000000002bb1c6e in std::_Function_handler<nebula::cpp2::ErrorCode (), std::_Bind<nebula::cpp2::ErrorCode (nebula::storage::RebuildIndexTask::*(nebula::storage::RebuildIndexTask*, int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > >))(int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > > const&)> >::_M_invoke(std::_Any_data const&) (
    __functor=...) at /opt/vesoft/toolset/gcc/9.1.0/include/c++/9.1.0/bits/std_function.h:285
#15 0x0000000002b8b72c in std::function<nebula::cpp2::ErrorCode ()>::operator()() const (
    this=0x7fe8553e7470) at /opt/vesoft/toolset/gcc/9.1.0/include/c++/9.1.0/bits/std_function.h:690
#16 0x0000000002b88c6c in nebula::storage::AdminSubTask::invoke (this=0x7fe8553e7470)
    at /home/hs.zhang/repos/nebula/src/storage/admin/AdminTask.h:29
#17 0x0000000002b8ad4f in nebula::storage::AdminTaskManager::runSubTask (
    this=0x5671200 <nebula::storage::AdminTaskManager::instance()::sAdminTaskManager>, handle=...)
    at /home/hs.zhang/repos/nebula/src/storage/admin/AdminTaskManager.cpp:163
#18 0x0000000002b9b94e in std::__invoke_impl<void, void (nebula::storage::AdminTaskManager::*&)(std::pair<int, int>), nebula::storage::AdminTaskManager*&, std::pair<int, int>&> (__f=
    @0x7fe85c20e0c0: (void (nebula::storage::AdminTaskManager::*)(nebula::storage::AdminTaskManager * const, std::pair<int, int>)) 0x2b8ab24 <nebula::storage::AdminTaskManager::runSubTask(std::pair<int, int>)>, __t=@0x7fe85c20e0d8: 0x5671200 <nebula::storage::AdminTaskManager::instance()::sAdminTaskManager>, 
    __args#0=...) at /opt/vesoft/toolset/gcc/9.1.0/include/c++/9.1.0/bits/invoke.h:73
#19 0x0000000002b9a725 in std::__invoke<void (nebula::storage::AdminTaskManager::*&)(std::pair<int, int>), nebula::storage::AdminTaskManager*&, std::pair<int, int>&> (__fn=
    @0x7fe85c20e0c0: (void (nebula::storage::AdminTaskManager::*)(nebula::storage::AdminTaskManager * const, std::pair<int, int>)) 0x2b8ab24 <nebula::storage::AdminTaskManager::runSubTask(std::pair<int, int>)>, 
    __args#0=@0x7fe85c20e0d8: 0x5671200 <nebula::storage::AdminTaskManager::instance()::sAdminTaskManager>, __args#1=...) at /opt/vesoft/toolset/gcc/9.1.0/include/c++/9.1.0/bits/invoke.h:95
#20 0x0000000002b985ff in std::_Bind<void (nebula::storage::AdminTaskManager::*(nebula::storage::AdminTaskManager*, std::pair<int, int>))(std::pair<int, int>)>::__call<void, , 0ul, 1ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) (this=0x7fe85c20e0c0, __args=...)
    at /opt/vesoft/toolset/gcc/9.1.0/include/c++/9.1.0/functional:400
#21 0x0000000002b9537e in std::_Bind<void (nebula::storage::AdminTaskManager::*(nebula::storage::AdminTaskManager*, std::pair<int, int>))(std::pair<int, int>)>::operator()<, void>() (this=0x7fe85c20e0c0)
--Type <RET> for more, q to quit, c to continue without paging--
    at /opt/vesoft/toolset/gcc/9.1.0/include/c++/9.1.0/functional:484
#22 0x0000000002b92a13 in folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (nebula::storage::AdminTaskManager::*(nebula::storage::AdminTaskManager*, std::pair<int, int>))(std::pair<int, int>)> >(folly::detail::function::Data&) (p=...)
    at /opt/vesoft/third-party/2.0/include/folly/Function.h:385
#23 0x0000000004341d4d in folly::ThreadPoolExecutor::runTask(std::shared_ptr<folly::ThreadPoolExecutor::Thread> const&, folly::ThreadPoolExecutor::Task&&) ()
#24 0x00000000043362ef in ?? ()
#25 0x00000000043a5e0b in folly::EventBase::FunctionLoopCallback::runLoopCallback() ()
#26 0x00000000043a065a in folly::EventBase::runLoopCallbacks() ()
#27 0x00000000043a1ff2 in folly::EventBase::loopBody(int, bool) ()
#28 0x00000000043a2aaf in folly::EventBase::loop() ()
#29 0x00000000043a4bb4 in folly::EventBase::loopForever() ()
#30 0x0000000004336c79 in folly::IOThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>) ()
#31 0x00000000043431d7 in void folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) ()
#32 0x0000000002a09dfd in folly::detail::function::FunctionTraits<void ()>::operator()() (
    this=0x7fe871231010) at /opt/vesoft/third-party/2.0/include/folly/Function.h:400
#33 0x0000000002a1770e in folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}::operator()() (__closure=0x7fe871231010)
    at /opt/vesoft/third-party/2.0/include/folly/executors/thread_factory/NamedThreadFactory.h:40
#34 0x0000000002a5aebc in std::__invoke_impl<void, folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}>(std::__invoke_other, folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}&&) (__f=...) at /opt/vesoft/toolset/gcc/9.1.0/include/c++/9.1.0/bits/invoke.h:60
#35 0x0000000002a5a6e8 in std::__invoke<folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}>(std::__invoke_result&&, (folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}&&)...) (__fn=...) at /opt/vesoft/toolset/gcc/9.1.0/include/c++/9.1.0/bits/invoke.h:95
#36 0x0000000002a59e6e in std::thread::_Invoker<std::tuple<folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x7fe871231010)
    at /opt/vesoft/toolset/gcc/9.1.0/include/c++/9.1.0/thread:244
#37 0x0000000002a59780 in std::thread::_Invoker<std::tuple<folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}> >::operator()() (this=0x7fe871231010)
    at /opt/vesoft/toolset/gcc/9.1.0/include/c++/9.1.0/thread:251
#38 0x0000000002a57100 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<folly::NamedThreadFa--Type <RET> for more, q to quit, c to continue without paging--
ctory::newThread(folly::Function<void ()>&&)::{lambda()#1}> > >::_M_run() (this=0x7fe871231000)
    at /opt/vesoft/toolset/gcc/9.1.0/include/c++/9.1.0/thread:195
#39 0x0000000004875e90 in execute_native_thread_routine ()
#40 0x00007fe8a110aea5 in start_thread () from /lib64/libpthread.so.0
#41 0x00007fe8a0e339fd in clone () from /lib64/libc.so.6

@Shylock-Hg
Copy link
Contributor Author

I get another different stack.It is abort by LOG(FATAL).But I am not sure if there is any relationship with the above mentioned problem.

#0  0x00007fe8a0d6b387 in raise () from /lib64/libc.so.6
#1  0x00007fe8a0d6ca78 in abort () from /lib64/libc.so.6
#2  0x0000000004430e7d in google::LogMessage::Fail() ()
#3  0x0000000004435c5e in google::LogMessage::SendToLog() ()
#4  0x0000000004430b4e in google::LogMessage::Flush() ()
#5  0x00000000044313a9 in google::LogMessageFatal::~LogMessageFatal() ()
#6  0x0000000002bab0f9 in nebula::kvstore::RateLimiter::consume (this=0x7fe85a849460, spaceId=77, 
    partId=3, toConsume=0) at /home/hs.zhang/repos/nebula/src/kvstore/RateLimiter.h:43
#7  0x0000000002bae1ea in nebula::storage::RebuildIndexTask::writeOperation (this=0x7fe89186a400, 
    space=77, part=3, batchHolder=0x7fe85c20e300)
    at /home/hs.zhang/repos/nebula/src/storage/admin/RebuildIndexTask.cpp:243
#8  0x0000000002bad537 in nebula::storage::RebuildIndexTask::buildIndexOnOperations (
    this=0x7fe89186a400, space=77, part=3)
--Type <RET> for more, q to quit, c to continue without paging--
   Task.cpp:166
#9  0x0000000002bac92c in nebula::storage::RebuildIndexTask::invoke (this=0x7fe89186a400, space=77, 
    part=3, items=...) at /home/hs.zhang/repos/nebula/src/storage/admin/RebuildIndexTask.cpp:109
#10 0x0000000002bba0db in std::__invoke_impl<nebula::cpp2::ErrorCode, nebula::cpp2::ErrorCode (nebula::storage::RebuildIndexTask::*&)(int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > > const&), nebula::storage::RebuildIndexTask*&, int&, int&, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > >&> (__f=
    @0x7fe871226700: (nebula::cpp2::ErrorCode (nebula::storage::RebuildIndexTask::*)(nebula::storage::RebuildIndexTask * const, int, int, const std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > > &)) 0x2bac2fc <nebula::storage::RebuildIndexTask::invoke(int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > > const&)>, __t=@0x7fe871226730: 0x7fe89186a400, 
    __args#0=@0x7fe87122672c: 77, __args#1=@0x7fe871226728: 3, __args#2=...)
    at /opt/vesoft/toolset/gcc/9.1.0/include/c++/9.1.0/bits/invoke.h:73
#11 0x0000000002bb7f3d in std::__invoke<nebula::cpp2::ErrorCode (nebula::storage::RebuildIndexTask::*&)(int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > > const&), nebula::storage::RebuildIndexTask*&, int&, int&, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > >&> (__fn=
    @0x7fe871226700: (nebula::cpp2::ErrorCode (nebula::storage::RebuildIndexTask::*)(nebula::storage::RebuildIndexTask * const, int, int, const std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > > &)) 0x2bac2fc <nebula::storage::RebuildIndexTask::invoke(int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > > const&)>, __args#0=@0x7fe871226730: 0x7fe89186a400, 
    __args#1=@0x7fe87122672c: 77, __args#2=@0x7fe871226728: 3, __args#3=...)
    at /opt/vesoft/toolset/gcc/9.1.0/include/c++/9.1.0/bits/invoke.h:95
#12 0x0000000002bb61d8 in std::_Bind<nebula::cpp2::ErrorCode (nebula::storage::RebuildIndexTask::*(nebula::storage::RebuildIndexTask*, int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > >))(int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > > const&)>::__call<nebula::cpp2::ErrorCode, , 0ul, 1ul, 2ul, 3ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul, 2ul, 3ul>) (this=0x7fe871226700, __args=...)
    at /opt/vesoft/toolset/gcc/9.1.0/include/c++/9.1.0/functional:400
#13 0x0000000002bb3eb8 in std::_Bind<nebula::cpp2::ErrorCode (nebula::storage::RebuildIndexTask::*(nebula::storage::RebuildIndexTask*, int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std--Type <RET> for more, q to quit, c to continue without paging--
::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > >))(int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > > const&)>::operator()<, nebula::cpp2::ErrorCode>() (this=0x7fe871226700)
    at /opt/vesoft/toolset/gcc/9.1.0/include/c++/9.1.0/functional:484
#14 0x0000000002bb1c6e in std::_Function_handler<nebula::cpp2::ErrorCode (), std::_Bind<nebula::cpp2::ErrorCode (nebula::storage::RebuildIndexTask::*(nebula::storage::RebuildIndexTask*, int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > >))(int, int, std::vector<std::shared_ptr<nebula::meta::cpp2::IndexItem>, std::allocator<std::shared_ptr<nebula::meta::cpp2::IndexItem> > > const&)> >::_M_invoke(std::_Any_data const&) (
    __functor=...) at /opt/vesoft/toolset/gcc/9.1.0/include/c++/9.1.0/bits/std_function.h:285
#15 0x0000000002b8b72c in std::function<nebula::cpp2::ErrorCode ()>::operator()() const (
    this=0x7fe8553e7470) at /opt/vesoft/toolset/gcc/9.1.0/include/c++/9.1.0/bits/std_function.h:690
#16 0x0000000002b88c6c in nebula::storage::AdminSubTask::invoke (this=0x7fe8553e7470)
    at /home/hs.zhang/repos/nebula/src/storage/admin/AdminTask.h:29
#17 0x0000000002b8ad4f in nebula::storage::AdminTaskManager::runSubTask (
    this=0x5671200 <nebula::storage::AdminTaskManager::instance()::sAdminTaskManager>, handle=...)
    at /home/hs.zhang/repos/nebula/src/storage/admin/AdminTaskManager.cpp:163
#18 0x0000000002b9b94e in std::__invoke_impl<void, void (nebula::storage::AdminTaskManager::*&)(std::pair<int, int>), nebula::storage::AdminTaskManager*&, std::pair<int, int>&> (__f=
    @0x7fe85c20e0c0: (void (nebula::storage::AdminTaskManager::*)(nebula::storage::AdminTaskManager * const, std::pair<int, int>)) 0x2b8ab24 <nebula::storage::AdminTaskManager::runSubTask(std::pair<int, int>)>, __t=@0x7fe85c20e0d8: 0x5671200 <nebula::storage::AdminTaskManager::instance()::sAdminTaskManager>, 
    __args#0=...) at /opt/vesoft/toolset/gcc/9.1.0/include/c++/9.1.0/bits/invoke.h:73
#19 0x0000000002b9a725 in std::__invoke<void (nebula::storage::AdminTaskManager::*&)(std::pair<int, int>), nebula::storage::AdminTaskManager*&, std::pair<int, int>&> (__fn=
    @0x7fe85c20e0c0: (void (nebula::storage::AdminTaskManager::*)(nebula::storage::AdminTaskManager * const, std::pair<int, int>)) 0x2b8ab24 <nebula::storage::AdminTaskManager::runSubTask(std::pair<int, int>)>, 
    __args#0=@0x7fe85c20e0d8: 0x5671200 <nebula::storage::AdminTaskManager::instance()::sAdminTaskManager>, __args#1=...) at /opt/vesoft/toolset/gcc/9.1.0/include/c++/9.1.0/bits/invoke.h:95
#20 0x0000000002b985ff in std::_Bind<void (nebula::storage::AdminTaskManager::*(nebula::storage::AdminTaskManager*, std::pair<int, int>))(std::pair<int, int>)>::__call<void, , 0ul, 1ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) (this=0x7fe85c20e0c0, __args=...)
    at /opt/vesoft/toolset/gcc/9.1.0/include/c++/9.1.0/functional:400
#21 0x0000000002b9537e in std::_Bind<void (nebula::storage::AdminTaskManager::*(nebula::storage::AdminTaskManager*, std::pair<int, int>))(std::pair<int, int>)>::operator()<, void>() (this=0x7fe85c20e0c0)
--Type <RET> for more, q to quit, c to continue without paging--
    at /opt/vesoft/toolset/gcc/9.1.0/include/c++/9.1.0/functional:484
#22 0x0000000002b92a13 in folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (nebula::storage::AdminTaskManager::*(nebula::storage::AdminTaskManager*, std::pair<int, int>))(std::pair<int, int>)> >(folly::detail::function::Data&) (p=...)
    at /opt/vesoft/third-party/2.0/include/folly/Function.h:385
#23 0x0000000004341d4d in folly::ThreadPoolExecutor::runTask(std::shared_ptr<folly::ThreadPoolExecutor::Thread> const&, folly::ThreadPoolExecutor::Task&&) ()
#24 0x00000000043362ef in ?? ()
#25 0x00000000043a5e0b in folly::EventBase::FunctionLoopCallback::runLoopCallback() ()
#26 0x00000000043a065a in folly::EventBase::runLoopCallbacks() ()
#27 0x00000000043a1ff2 in folly::EventBase::loopBody(int, bool) ()
#28 0x00000000043a2aaf in folly::EventBase::loop() ()
#29 0x00000000043a4bb4 in folly::EventBase::loopForever() ()
#30 0x0000000004336c79 in folly::IOThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>) ()
#31 0x00000000043431d7 in void folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) ()
#32 0x0000000002a09dfd in folly::detail::function::FunctionTraits<void ()>::operator()() (
    this=0x7fe871231010) at /opt/vesoft/third-party/2.0/include/folly/Function.h:400
#33 0x0000000002a1770e in folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}::operator()() (__closure=0x7fe871231010)
    at /opt/vesoft/third-party/2.0/include/folly/executors/thread_factory/NamedThreadFactory.h:40
#34 0x0000000002a5aebc in std::__invoke_impl<void, folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}>(std::__invoke_other, folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}&&) (__f=...) at /opt/vesoft/toolset/gcc/9.1.0/include/c++/9.1.0/bits/invoke.h:60
#35 0x0000000002a5a6e8 in std::__invoke<folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}>(std::__invoke_result&&, (folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}&&)...) (__fn=...) at /opt/vesoft/toolset/gcc/9.1.0/include/c++/9.1.0/bits/invoke.h:95
#36 0x0000000002a59e6e in std::thread::_Invoker<std::tuple<folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x7fe871231010)
    at /opt/vesoft/toolset/gcc/9.1.0/include/c++/9.1.0/thread:244
#37 0x0000000002a59780 in std::thread::_Invoker<std::tuple<folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}> >::operator()() (this=0x7fe871231010)
    at /opt/vesoft/toolset/gcc/9.1.0/include/c++/9.1.0/thread:251
#38 0x0000000002a57100 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<folly::NamedThreadFa--Type <RET> for more, q to quit, c to continue without paging--
ctory::newThread(folly::Function<void ()>&&)::{lambda()#1}> > >::_M_run() (this=0x7fe871231000)
    at /opt/vesoft/toolset/gcc/9.1.0/include/c++/9.1.0/thread:195
#39 0x0000000004875e90 in execute_native_thread_routine ()
#40 0x00007fe8a110aea5 in start_thread () from /lib64/libpthread.so.0
#41 0x00007fe8a0e339fd in clone () from /lib64/libc.so.6

It's same with #2761

@critical27 critical27 mentioned this issue Sep 7, 2021
Merged
@yixinglu yixinglu linked a pull request Sep 7, 2021 that will close this issue
skip the wait in rate limiter in CI #2795
Merged
@jamieliu1023 jamieliu1023 mentioned this issue Sep 11, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@critical27 critical27

Labels
type/bug Type: something is unexpected
Projects
None yet
Milestone
v2.6.0
Development

Successfully merging a pull request may close this issue.

skip the wait in rate limiter in CI critical27/nebula
4 participants
@critical27 @cangfengzhs @Shylock-Hg @Sophie-Xie