check session when applying operation #15

KarlManong · 2019-01-09T11:24:16Z

No description provided.

adembo · 2019-01-09T19:01:54Z

Thank you for this patch.

The Kudu project doesn't accept GitHub pull requests; we do code review and submission via the gerrit tool. Could you please adapt this PR into a gerrit change? Instructions on how to use gerrit are available here.

KarlManong · 2019-01-10T01:34:51Z

Sorry, I have some problems when using gerrit, which I can't fix:
ssh: connect to host gerrit.cloudera.org port 29418: Connection timed out

adembo · 2019-01-11T05:22:31Z

If the timeout is while fetching the commit hook needed by gerrit, does the following work for you?

cd kudu
gitdir=$(git rev-parse --git-dir)
curl -LSsf https://gerrit.cloudera.org/tools/hooks/commit-msg -o ${gitdir}/hooks/commit-msg
chmod +x ${gitdir}/hooks/commit-msg

If it's when trying to access gerrit's repo (i.e. git pull or git push), one of us can create the gerrit change on your behalf.

KarlManong · 2019-01-11T07:14:40Z

I have a problem with my development computer, so I can't do it. Please fix it in your way.

adembo · 2019-01-18T04:35:38Z

OK, I made a gerrit change here.

Doesn't look like I have permission to close this pull request yet; I'll work on that.

granthenke · 2019-01-18T05:22:49Z

I can mark as closed. I will also review the patch @adembo created.

Noticed this on the flaky test dashboard for alter_table-randomized-test; see the end of the commit message for the complete output. I also removed an unrelated and unnecessary lock acquisition. To test, I looped alter_table_randomized-test in slow mode with TSAN and the two failures I saw did not report any data races. WARNING: ThreadSanitizer: data race (pid=17016) Read of size 8 at 0x7b4c000010d0 by thread T68 (mutexes: write M1500): #0 std::__1::unique_ptr<kudu::hms::HmsCatalog, std::__1::default_delete<kudu::hms::HmsCatalog> >::operator bool() const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:2583:19 (libmaster.so+0xb99b1) #1 kudu::master::CatalogManager::PrepareForLeadershipTask() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/catalog_manager.cc:1055 (libmaster.so+0xb99b1) #2 kudu::internal::RunnableAdapter<void (kudu::master::CatalogManager::*)()>::Run(kudu::master::CatalogManager*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:136:12 (libmaster.so+0x102fa9) #3 kudu::internal::InvokeHelper<false, void, kudu::internal::RunnableAdapter<void (kudu::master::CatalogManager::*)()>, void ()(kudu::master::CatalogManager*)>::MakeItSo(kudu::internal::RunnableAdapter<void (kudu::master::CatalogManager::*)()>, kudu::master::CatalogManager*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:873:14 (libmaster.so+0x102ec5) #4 kudu::internal::Invoker<1, kudu::internal::BindState<kudu::internal::RunnableAdapter<void (kudu::master::CatalogManager::*)()>, void ()(kudu::master::CatalogManager*), void ()(kudu::internal::UnretainedWrapper<kudu::master::CatalogManager>)>, void ()(kudu::master::CatalogManager*)>::Run(kudu::internal::BindStateBase*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:1065:12 (libmaster.so+0x102e0a) #5 kudu::Callback<void ()()>::Run() const /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/callback.h:396:12 (libconsensus.so+0xa6dfd) #6 kudu::ClosureRunnable::Run() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:76:9 (libkudu_util.so+0x1cc9ad) #7 kudu::ThreadPool::DispatchThread() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:686:22 (libkudu_util.so+0x1c86d8) #8 boost::_mfi::mf0<void, kudu::ThreadPool>::operator()(kudu::ThreadPool*) const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkudu_util.so+0x1d3649) #9 void boost::_bi::list1<boost::_bi::value<kudu::ThreadPool*> >::operator()<boost::_mfi::mf0<void, kudu::ThreadPool>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf0<void, kudu::ThreadPool>&, boost::_bi::list0&, int) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkudu_util.so+0x1d359a) #10 boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::ThreadPool>, boost::_bi::list1<boost::_bi::value<kudu::ThreadPool*> > >::operator()() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkudu_util.so+0x1d3523) #11 boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::ThreadPool>, boost::_bi::list1<boost::_bi::value<kudu::ThreadPool*> > >, void>::invoke(boost::detail::function::function_buffer&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkudu_util.so+0x1d3319) #12 boost::function0<void>::operator()() const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xb6651) #13 kudu::Thread::SuperviseThread(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:615:3 (libkudu_util.so+0x1bfe34) Previous write of size 8 at 0x7b4c000010d0 by thread T59: #0 std::__1::unique_ptr<kudu::hms::HmsCatalog, std::__1::default_delete<kudu::hms::HmsCatalog> >::reset(kudu::hms::HmsCatalog*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:2596:20 (libmaster.so+0xb8b6f) #1 kudu::master::CatalogManager::Init(bool) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/catalog_manager.cc:730 (libmaster.so+0xb8b6f) #2 kudu::master::Master::InitCatalogManager() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/master.cc:216:3 (libmaster.so+0x11fa5f) #3 kudu::master::Master::InitCatalogManagerTask() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/master.cc:205:14 (libmaster.so+0x11f8b2) #4 kudu::internal::RunnableAdapter<void (kudu::master::Master::*)()>::Run(kudu::master::Master*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:136:12 (libmaster.so+0x124449) #5 kudu::internal::InvokeHelper<false, void, kudu::internal::RunnableAdapter<void (kudu::master::Master::*)()>, void ()(kudu::master::Master*)>::MakeItSo(kudu::internal::RunnableAdapter<void (kudu::master::Master::*)()>, kudu::master::Master*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:873:14 (libmaster.so+0x124365) #6 kudu::internal::Invoker<1, kudu::internal::BindState<kudu::internal::RunnableAdapter<void (kudu::master::Master::*)()>, void ()(kudu::master::Master*), void ()(kudu::internal::UnretainedWrapper<kudu::master::Master>)>, void ()(kudu::master::Master*)>::Run(kudu::internal::BindStateBase*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:1065:12 (libmaster.so+0x1242aa) #7 kudu::Callback<void ()()>::Run() const /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/callback.h:396:12 (libconsensus.so+0xa6dfd) #8 kudu::ClosureRunnable::Run() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:76:9 (libkudu_util.so+0x1cc9ad) #9 kudu::ThreadPool::DispatchThread() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:686:22 (libkudu_util.so+0x1c86d8) #10 boost::_mfi::mf0<void, kudu::ThreadPool>::operator()(kudu::ThreadPool*) const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkudu_util.so+0x1d3649) #11 void boost::_bi::list1<boost::_bi::value<kudu::ThreadPool*> >::operator()<boost::_mfi::mf0<void, kudu::ThreadPool>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf0<void, kudu::ThreadPool>&, boost::_bi::list0&, int) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkudu_util.so+0x1d359a) #12 boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::ThreadPool>, boost::_bi::list1<boost::_bi::value<kudu::ThreadPool*> > >::operator()() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkudu_util.so+0x1d3523) #13 boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::ThreadPool>, boost::_bi::list1<boost::_bi::value<kudu::ThreadPool*> > >, void>::invoke(boost::detail::function::function_buffer&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkudu_util.so+0x1d3319) #14 boost::function0<void>::operator()() const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xb6651) #15 kudu::Thread::SuperviseThread(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:615:3 (libkudu_util.so+0x1bfe34) Location is heap block of size 432 at 0x7b4c00000fc0 allocated by main thread: #0 operator new(unsigned long) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:57 (kudu-master+0x4c84a3) #1 kudu::master::Master::Master(kudu::master::MasterOptions const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/master.cc:122:22 (libmaster.so+0x11e3d5) #2 kudu::master::MasterMain(int, char**) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/master_main.cc:79:10 (kudu-master+0x4cb4de) #3 main /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/master_main.cc:98:10 (kudu-master+0x4cb1be) Mutex M1500 (0x7b4c00001100) created at: #0 pthread_rwlock_init /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1304 (kudu-master+0x4593b4) #1 kudu::RWMutex::Init(kudu::RWMutex::Priority) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/rw_mutex.cc:78:8 (libkudu_util.so+0x1acad8) #2 kudu::RWMutex::RWMutex(kudu::RWMutex::Priority) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/rw_mutex.cc:56:3 (libkudu_util.so+0x1acd13) #3 kudu::master::CatalogManager::CatalogManager(kudu::master::Master*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/catalog_manager.cc:688:7 (libmaster.so+0xb81d7) #4 kudu::master::Master::Master(kudu::master::MasterOptions const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/master.cc:122:26 (libmaster.so+0x11e3e3) #5 kudu::master::MasterMain(int, char**) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/master_main.cc:79:10 (kudu-master+0x4cb4de) #6 main /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/master_main.cc:98:10 (kudu-master+0x4cb1be) Thread T68 'leader-initiali' (tid=17094, running) created by thread T65 at: #0 pthread_create /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:992 (kudu-master+0x45af0b) #1 kudu::Thread::StartThread(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, boost::function<void ()()> const&, unsigned long, scoped_refptr<kudu::Thread>*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:559:15 (libkudu_util.so+0x1bf61b) #2 kudu::Status kudu::Thread::Create<void (kudu::ThreadPool::*)(), kudu::ThreadPool*>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, void (kudu::ThreadPool::* const&)(), kudu::ThreadPool* const&, scoped_refptr<kudu::Thread>*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.h:164:12 (libkudu_util.so+0x1ca9f5) #3 kudu::ThreadPool::CreateThread() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:749:10 (libkudu_util.so+0x1c7ce2) #4 kudu::ThreadPool::DoSubmit(std::__1::shared_ptr<kudu::Runnable>, kudu::ThreadPoolToken*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:556:21 (libkudu_util.so+0x1c64af) #5 kudu::ThreadPool::Submit(std::__1::shared_ptr<kudu::Runnable>) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:458:10 (libkudu_util.so+0x1c7f4f) #6 kudu::ThreadPool::SubmitClosure(kudu::Callback<void ()()>) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:450:10 (libkudu_util.so+0x1c7e91) #7 kudu::master::CatalogManager::ElectedAsLeaderCb() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/catalog_manager.cc:754:33 (libmaster.so+0xb936b) #8 kudu::internal::RunnableAdapter<kudu::Status (kudu::master::CatalogManager::*)()>::Run(kudu::master::CatalogManager*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:136:12 (libmaster.so+0x104130) #9 kudu::internal::InvokeHelper<false, kudu::Status, kudu::internal::RunnableAdapter<kudu::Status (kudu::master::CatalogManager::*)()>, void ()(kudu::master::CatalogManager*)>::MakeItSo(kudu::internal::RunnableAdapter<kudu::Status (kudu::master::CatalogManager::*)()>, kudu::master::CatalogManager*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:865:21 (libmaster.so+0x10409d) #10 kudu::internal::Invoker<1, kudu::internal::BindState<kudu::internal::RunnableAdapter<kudu::Status (kudu::master::CatalogManager::*)()>, kudu::Status ()(kudu::master::CatalogManager*), void ()(kudu::internal::UnretainedWrapper<kudu::master::CatalogManager>)>, kudu::Status ()(kudu::master::CatalogManager*)>::Run(kudu::internal::BindStateBase*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:1065:12 (libmaster.so+0x10400f) #11 kudu::Callback<kudu::Status ()()>::Run() const /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/callback.h:396:12 (libmaster.so+0x14bed6) #12 kudu::master::SysCatalogTable::SysCatalogStateChanged(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/sys_catalog.cc:343:27 (libmaster.so+0x145539) #13 kudu::internal::RunnableAdapter<void (kudu::master::SysCatalogTable::*)(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>::Run(kudu::master::SysCatalogTable*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:250:12 (libmaster.so+0x15269d) #14 kudu::internal::InvokeHelper<false, void, kudu::internal::RunnableAdapter<void (kudu::master::SysCatalogTable::*)(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>, void ()(kudu::master::SysCatalogTable*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>::MakeItSo(kudu::internal::RunnableAdapter<void (kudu::master::SysCatalogTable::*)(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>, kudu::master::SysCatalogTable*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:907:14 (libmaster.so+0x15256b) #15 kudu::internal::Invoker<2, kudu::internal::BindState<kudu::internal::RunnableAdapter<void (kudu::master::SysCatalogTable::*)(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>, void ()(kudu::master::SysCatalogTable*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&), void ()(kudu::internal::UnretainedWrapper<kudu::master::SysCatalogTable>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)>, void ()(kudu::master::SysCatalogTable*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>::Run(kudu::internal::BindStateBase*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:1242:12 (libmaster.so+0x152459) #16 kudu::Callback<void ()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>::Run(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/callback.h:436:12 (libtablet.so+0x151681) #17 kudu::internal::InvokeHelper<false, void, kudu::Callback<void ()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>, void ()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>::MakeItSo(kudu::Callback<void ()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:873:14 (libconsensus.so+0xea428) #18 kudu::internal::Invoker<1, kudu::internal::BindState<kudu::Callback<void ()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>, void ()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&), void ()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)>, void ()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>::Run(kudu::internal::BindStateBase*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:1065:12 (libconsensus.so+0xea3c3) #19 kudu::Callback<void ()()>::Run() const /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/callback.h:396:12 (libconsensus.so+0xa6dfd) #20 kudu::ClosureRunnable::Run() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:76:9 (libkudu_util.so+0x1cc9ad) #21 kudu::ThreadPool::DispatchThread() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:686:22 (libkudu_util.so+0x1c86d8) #22 boost::_mfi::mf0<void, kudu::ThreadPool>::operator()(kudu::ThreadPool*) const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkudu_util.so+0x1d3649) #23 void boost::_bi::list1<boost::_bi::value<kudu::ThreadPool*> >::operator()<boost::_mfi::mf0<void, kudu::ThreadPool>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf0<void, kudu::ThreadPool>&, boost::_bi::list0&, int) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkudu_util.so+0x1d359a) #24 boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::ThreadPool>, boost::_bi::list1<boost::_bi::value<kudu::ThreadPool*> > >::operator()() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkudu_util.so+0x1d3523) #25 boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::ThreadPool>, boost::_bi::list1<boost::_bi::value<kudu::ThreadPool*> > >, void>::invoke(boost::detail::function::function_buffer&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkudu_util.so+0x1d3319) #26 boost::function0<void>::operator()() const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xb6651) #27 kudu::Thread::SuperviseThread(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:615:3 (libkudu_util.so+0x1bfe34) Thread T59 'init [worker]-1' (tid=17081, running) created by main thread at: #0 pthread_create /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:992 (kudu-master+0x45af0b) #1 kudu::Thread::StartThread(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, boost::function<void ()()> const&, unsigned long, scoped_refptr<kudu::Thread>*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:559:15 (libkudu_util.so+0x1bf61b) #2 kudu::Status kudu::Thread::Create<void (kudu::ThreadPool::*)(), kudu::ThreadPool*>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, void (kudu::ThreadPool::* const&)(), kudu::ThreadPool* const&, scoped_refptr<kudu::Thread>*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.h:164:12 (libkudu_util.so+0x1ca9f5) #3 kudu::ThreadPool::CreateThread() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:749:10 (libkudu_util.so+0x1c7ce2) #4 kudu::ThreadPool::DoSubmit(std::__1::shared_ptr<kudu::Runnable>, kudu::ThreadPoolToken*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:556:21 (libkudu_util.so+0x1c64af) #5 kudu::ThreadPool::Submit(std::__1::shared_ptr<kudu::Runnable>) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:458:10 (libkudu_util.so+0x1c7f4f) #6 kudu::ThreadPool::SubmitClosure(kudu::Callback<void ()()>) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:450:10 (libkudu_util.so+0x1c7e91) #7 kudu::master::Master::StartAsync() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/master.cc:196:3 (libmaster.so+0x11f260) #8 kudu::master::Master::Start() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/master.cc:170:3 (libmaster.so+0x11ef25) #9 kudu::master::MasterMain(int, char**) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/master_main.cc:84:3 (kudu-master+0x4cb584) #10 main /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/master_main.cc:98:10 (kudu-master+0x4cb1be) Change-Id: I090a832b7fb25d8cb1e770c025048f73ac997eac Reviewed-on: http://gerrit.cloudera.org:8080/11818 Tested-by: Kudu Jenkins Reviewed-by: Alexey Serbin <[email protected]> Reviewed-by: Hao Hao <[email protected]>

In real environments there are various types of failures that are "expected". In particular, it's not unlikely for the client to occasionally connect to a server which has gone down or which is not the leader for a given tablet anymore. In that case, we should make sure that the logging output is easy to read and doesn't include irrelevant stack traces stemming from Netty internals. This patch addresses several of those cases. In particular, it gets rid of "unexpected exception" messages (and associated stacks) in the case of "connection refused" errors. It also helpfully includes the underlying error when invalidating the tablet cache for a location. There's a relatively simple new unit test, and I also ran some before/after comparisons using the "loadgen" sample from the kudu-examples repo against a local cluster with one TS: Case 1: Master down when we start the client: ============================================================ Before the patch -------------- [New I/O worker apache#12] WARN org.apache.kudu.client.ConnectToCluster - Error receiving response from localhost:7051 org.apache.kudu.client.RecoverableException: connection closed at org.apache.kudu.client.Connection.channelClosed(Connection.java:254) at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:88) at org.apache.kudu.client.Connection.handleUpstream(Connection.java:238) at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.apache.kudu.shaded.org.jboss.netty.handler.timeout.ReadTimeoutHandler.channelClosed(ReadTimeoutHandler.java:176) at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:88) at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.apache.kudu.shaded.org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:60) at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.apache.kudu.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:493) at org.apache.kudu.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.channelClosed(FrameDecoder.java:371) at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:88) at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.apache.kudu.shaded.org.jboss.netty.channel.Channels.fireChannelClosed(Channels.java:468) at org.apache.kudu.shaded.org.jboss.netty.channel.Channels$6.run(Channels.java:457) at org.apache.kudu.shaded.org.jboss.netty.channel.socket.ChannelRunnableWrapper.run(ChannelRunnableWrapper.java:40) at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:391) at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:315) at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.apache.kudu.shaded.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.apache.kudu.shaded.org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) [New I/O boss apache#17] ERROR org.apache.kudu.client.Connection - [peer master-localhost:7051(localhost:7051)] unexpected exception from downstream on [id: 0xf9759051] java.net.ConnectException: Connection refused: localhost/127.0.0.1:7051 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152) at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105) at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79) at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42) at org.apache.kudu.shaded.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.apache.kudu.shaded.org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) [New I/O worker apache#12] WARN org.apache.kudu.client.ConnectToCluster - Unable to find the leader master localhost:7051; will retry ... repeats with exponential backoff After the patch -------------- [New I/O boss apache#17] INFO org.apache.kudu.client.Connection - Failed to connect to peer master-localhost:7051(localhost:7051): Connection refused: localhost/127.0.0.1:7051 [New I/O worker apache#1] INFO org.apache.kudu.client.ConnectToCluster - Unable to connect to master localhost:7051: Connection refused: localhost/127.0.0.1:7051 [New I/O worker apache#1] WARN org.apache.kudu.client.ConnectToCluster - Unable to find the leader master localhost:7051; will retry ... repeats with exponential backoff Case 2: Tserver is down ============================================================ Before the patch ---------------- [New I/O worker apache#11] INFO org.apache.kudu.client.AsyncKuduClient - Removing server fc9f6ab955cc47a8ab3653d0170305b0 from this tablet's cache 9da164fbcd93404ea8e38e9491eb3fa8 [New I/O boss apache#17] ERROR org.apache.kudu.client.Connection - [peer fc9f6ab955cc47a8ab3653d0170305b0(todd-laptop:7050)] unexpected exception from downstream on [id: 0x270e3a82] java.net.ConnectException: Connection refused: todd-laptop/127.0.1.1:7050 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152) at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105) at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79) at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42) at org.apache.kudu.shaded.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.apache.kudu.shaded.org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ... repeats with exponential backoff After the patch --------------- [New I/O boss apache#17] INFO org.apache.kudu.client.Connection - Failed to connect to peer fc9f6ab955cc47a8ab3653d0170305b0(todd-laptop:7050): Connection refused: todd-laptop/127.0.1.1:7050 [New I/O worker apache#15] INFO org.apache.kudu.client.AsyncKuduClient - Invalidating location fc9f6ab955cc47a8ab3653d0170305b0(todd-laptop:7050) for tablet 478ddbaafb5b494ea0de8e2d0ed45a00: Connection refused: todd-laptop/127.0.1.1:7050 ... repeats with exponential backoff Case 3: A tablet is bad ============================================================ In this case I manually delete a non-replicated tablet that is being written to using 'kudu remote_replica delete'. This is unrecoverable since there is no other replica. Before the patch ----------------- [New I/O worker apache#10] INFO org.apache.kudu.client.AsyncKuduClient - Removing server fc9f6ab955cc47a8ab3653d0170305b0 from this tablet's cache a633f4a0c14b4404a8ae1f825b378867 ... repeats with exponential backoff After the patch ---------------- [New I/O worker apache#9] INFO org.apache.kudu.client.AsyncKuduClient - Invalidating location 36622c46b25f4da9ada43f8591728053(todd-laptop:7050) for tablet c0b9488597e8447595ad0ee0a3378f95: Tablet not RUNNING: STOPPED ... repeats with exponential backoff Change-Id: I4b8be871693fecc1ee46a4e238dd2ed8f0730d4b Reviewed-on: http://gerrit.cloudera.org:8080/9644 Reviewed-by: Grant Henke <[email protected]> Tested-by: Grant Henke <[email protected]>

Noticed this on the flaky test dashboard for alter_table-randomized-test; see the end of the commit message for the complete output. I also removed an unrelated and unnecessary lock acquisition. To test, I looped alter_table_randomized-test in slow mode with TSAN and the two failures I saw did not report any data races. WARNING: ThreadSanitizer: data race (pid=17016) Read of size 8 at 0x7b4c000010d0 by thread T68 (mutexes: write M1500): #0 std::__1::unique_ptr<kudu::hms::HmsCatalog, std::__1::default_delete<kudu::hms::HmsCatalog> >::operator bool() const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:2583:19 (libmaster.so+0xb99b1) apache#1 kudu::master::CatalogManager::PrepareForLeadershipTask() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/catalog_manager.cc:1055 (libmaster.so+0xb99b1) apache#2 kudu::internal::RunnableAdapter<void (kudu::master::CatalogManager::*)()>::Run(kudu::master::CatalogManager*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:136:12 (libmaster.so+0x102fa9) apache#3 kudu::internal::InvokeHelper<false, void, kudu::internal::RunnableAdapter<void (kudu::master::CatalogManager::*)()>, void ()(kudu::master::CatalogManager*)>::MakeItSo(kudu::internal::RunnableAdapter<void (kudu::master::CatalogManager::*)()>, kudu::master::CatalogManager*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:873:14 (libmaster.so+0x102ec5) apache#4 kudu::internal::Invoker<1, kudu::internal::BindState<kudu::internal::RunnableAdapter<void (kudu::master::CatalogManager::*)()>, void ()(kudu::master::CatalogManager*), void ()(kudu::internal::UnretainedWrapper<kudu::master::CatalogManager>)>, void ()(kudu::master::CatalogManager*)>::Run(kudu::internal::BindStateBase*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:1065:12 (libmaster.so+0x102e0a) apache#5 kudu::Callback<void ()()>::Run() const /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/callback.h:396:12 (libconsensus.so+0xa6dfd) apache#6 kudu::ClosureRunnable::Run() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:76:9 (libkudu_util.so+0x1cc9ad) apache#7 kudu::ThreadPool::DispatchThread() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:686:22 (libkudu_util.so+0x1c86d8) apache#8 boost::_mfi::mf0<void, kudu::ThreadPool>::operator()(kudu::ThreadPool*) const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkudu_util.so+0x1d3649) apache#9 void boost::_bi::list1<boost::_bi::value<kudu::ThreadPool*> >::operator()<boost::_mfi::mf0<void, kudu::ThreadPool>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf0<void, kudu::ThreadPool>&, boost::_bi::list0&, int) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkudu_util.so+0x1d359a) apache#10 boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::ThreadPool>, boost::_bi::list1<boost::_bi::value<kudu::ThreadPool*> > >::operator()() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkudu_util.so+0x1d3523) apache#11 boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::ThreadPool>, boost::_bi::list1<boost::_bi::value<kudu::ThreadPool*> > >, void>::invoke(boost::detail::function::function_buffer&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkudu_util.so+0x1d3319) apache#12 boost::function0<void>::operator()() const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xb6651) apache#13 kudu::Thread::SuperviseThread(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:615:3 (libkudu_util.so+0x1bfe34) Previous write of size 8 at 0x7b4c000010d0 by thread T59: #0 std::__1::unique_ptr<kudu::hms::HmsCatalog, std::__1::default_delete<kudu::hms::HmsCatalog> >::reset(kudu::hms::HmsCatalog*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:2596:20 (libmaster.so+0xb8b6f) apache#1 kudu::master::CatalogManager::Init(bool) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/catalog_manager.cc:730 (libmaster.so+0xb8b6f) apache#2 kudu::master::Master::InitCatalogManager() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/master.cc:216:3 (libmaster.so+0x11fa5f) apache#3 kudu::master::Master::InitCatalogManagerTask() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/master.cc:205:14 (libmaster.so+0x11f8b2) apache#4 kudu::internal::RunnableAdapter<void (kudu::master::Master::*)()>::Run(kudu::master::Master*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:136:12 (libmaster.so+0x124449) apache#5 kudu::internal::InvokeHelper<false, void, kudu::internal::RunnableAdapter<void (kudu::master::Master::*)()>, void ()(kudu::master::Master*)>::MakeItSo(kudu::internal::RunnableAdapter<void (kudu::master::Master::*)()>, kudu::master::Master*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:873:14 (libmaster.so+0x124365) apache#6 kudu::internal::Invoker<1, kudu::internal::BindState<kudu::internal::RunnableAdapter<void (kudu::master::Master::*)()>, void ()(kudu::master::Master*), void ()(kudu::internal::UnretainedWrapper<kudu::master::Master>)>, void ()(kudu::master::Master*)>::Run(kudu::internal::BindStateBase*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:1065:12 (libmaster.so+0x1242aa) apache#7 kudu::Callback<void ()()>::Run() const /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/callback.h:396:12 (libconsensus.so+0xa6dfd) apache#8 kudu::ClosureRunnable::Run() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:76:9 (libkudu_util.so+0x1cc9ad) apache#9 kudu::ThreadPool::DispatchThread() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:686:22 (libkudu_util.so+0x1c86d8) apache#10 boost::_mfi::mf0<void, kudu::ThreadPool>::operator()(kudu::ThreadPool*) const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkudu_util.so+0x1d3649) apache#11 void boost::_bi::list1<boost::_bi::value<kudu::ThreadPool*> >::operator()<boost::_mfi::mf0<void, kudu::ThreadPool>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf0<void, kudu::ThreadPool>&, boost::_bi::list0&, int) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkudu_util.so+0x1d359a) apache#12 boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::ThreadPool>, boost::_bi::list1<boost::_bi::value<kudu::ThreadPool*> > >::operator()() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkudu_util.so+0x1d3523) apache#13 boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::ThreadPool>, boost::_bi::list1<boost::_bi::value<kudu::ThreadPool*> > >, void>::invoke(boost::detail::function::function_buffer&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkudu_util.so+0x1d3319) apache#14 boost::function0<void>::operator()() const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xb6651) apache#15 kudu::Thread::SuperviseThread(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:615:3 (libkudu_util.so+0x1bfe34) Location is heap block of size 432 at 0x7b4c00000fc0 allocated by main thread: #0 operator new(unsigned long) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:57 (kudu-master+0x4c84a3) apache#1 kudu::master::Master::Master(kudu::master::MasterOptions const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/master.cc:122:22 (libmaster.so+0x11e3d5) apache#2 kudu::master::MasterMain(int, char**) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/master_main.cc:79:10 (kudu-master+0x4cb4de) apache#3 main /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/master_main.cc:98:10 (kudu-master+0x4cb1be) Mutex M1500 (0x7b4c00001100) created at: #0 pthread_rwlock_init /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1304 (kudu-master+0x4593b4) apache#1 kudu::RWMutex::Init(kudu::RWMutex::Priority) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/rw_mutex.cc:78:8 (libkudu_util.so+0x1acad8) apache#2 kudu::RWMutex::RWMutex(kudu::RWMutex::Priority) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/rw_mutex.cc:56:3 (libkudu_util.so+0x1acd13) apache#3 kudu::master::CatalogManager::CatalogManager(kudu::master::Master*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/catalog_manager.cc:688:7 (libmaster.so+0xb81d7) apache#4 kudu::master::Master::Master(kudu::master::MasterOptions const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/master.cc:122:26 (libmaster.so+0x11e3e3) apache#5 kudu::master::MasterMain(int, char**) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/master_main.cc:79:10 (kudu-master+0x4cb4de) apache#6 main /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/master_main.cc:98:10 (kudu-master+0x4cb1be) Thread T68 'leader-initiali' (tid=17094, running) created by thread T65 at: #0 pthread_create /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:992 (kudu-master+0x45af0b) apache#1 kudu::Thread::StartThread(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, boost::function<void ()()> const&, unsigned long, scoped_refptr<kudu::Thread>*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:559:15 (libkudu_util.so+0x1bf61b) apache#2 kudu::Status kudu::Thread::Create<void (kudu::ThreadPool::*)(), kudu::ThreadPool*>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, void (kudu::ThreadPool::* const&)(), kudu::ThreadPool* const&, scoped_refptr<kudu::Thread>*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.h:164:12 (libkudu_util.so+0x1ca9f5) apache#3 kudu::ThreadPool::CreateThread() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:749:10 (libkudu_util.so+0x1c7ce2) apache#4 kudu::ThreadPool::DoSubmit(std::__1::shared_ptr<kudu::Runnable>, kudu::ThreadPoolToken*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:556:21 (libkudu_util.so+0x1c64af) apache#5 kudu::ThreadPool::Submit(std::__1::shared_ptr<kudu::Runnable>) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:458:10 (libkudu_util.so+0x1c7f4f) apache#6 kudu::ThreadPool::SubmitClosure(kudu::Callback<void ()()>) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:450:10 (libkudu_util.so+0x1c7e91) apache#7 kudu::master::CatalogManager::ElectedAsLeaderCb() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/catalog_manager.cc:754:33 (libmaster.so+0xb936b) apache#8 kudu::internal::RunnableAdapter<kudu::Status (kudu::master::CatalogManager::*)()>::Run(kudu::master::CatalogManager*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:136:12 (libmaster.so+0x104130) apache#9 kudu::internal::InvokeHelper<false, kudu::Status, kudu::internal::RunnableAdapter<kudu::Status (kudu::master::CatalogManager::*)()>, void ()(kudu::master::CatalogManager*)>::MakeItSo(kudu::internal::RunnableAdapter<kudu::Status (kudu::master::CatalogManager::*)()>, kudu::master::CatalogManager*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:865:21 (libmaster.so+0x10409d) apache#10 kudu::internal::Invoker<1, kudu::internal::BindState<kudu::internal::RunnableAdapter<kudu::Status (kudu::master::CatalogManager::*)()>, kudu::Status ()(kudu::master::CatalogManager*), void ()(kudu::internal::UnretainedWrapper<kudu::master::CatalogManager>)>, kudu::Status ()(kudu::master::CatalogManager*)>::Run(kudu::internal::BindStateBase*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:1065:12 (libmaster.so+0x10400f) apache#11 kudu::Callback<kudu::Status ()()>::Run() const /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/callback.h:396:12 (libmaster.so+0x14bed6) apache#12 kudu::master::SysCatalogTable::SysCatalogStateChanged(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/sys_catalog.cc:343:27 (libmaster.so+0x145539) apache#13 kudu::internal::RunnableAdapter<void (kudu::master::SysCatalogTable::*)(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>::Run(kudu::master::SysCatalogTable*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:250:12 (libmaster.so+0x15269d) apache#14 kudu::internal::InvokeHelper<false, void, kudu::internal::RunnableAdapter<void (kudu::master::SysCatalogTable::*)(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>, void ()(kudu::master::SysCatalogTable*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>::MakeItSo(kudu::internal::RunnableAdapter<void (kudu::master::SysCatalogTable::*)(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>, kudu::master::SysCatalogTable*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:907:14 (libmaster.so+0x15256b) apache#15 kudu::internal::Invoker<2, kudu::internal::BindState<kudu::internal::RunnableAdapter<void (kudu::master::SysCatalogTable::*)(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>, void ()(kudu::master::SysCatalogTable*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&), void ()(kudu::internal::UnretainedWrapper<kudu::master::SysCatalogTable>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)>, void ()(kudu::master::SysCatalogTable*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>::Run(kudu::internal::BindStateBase*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:1242:12 (libmaster.so+0x152459) apache#16 kudu::Callback<void ()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>::Run(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/callback.h:436:12 (libtablet.so+0x151681) apache#17 kudu::internal::InvokeHelper<false, void, kudu::Callback<void ()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>, void ()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>::MakeItSo(kudu::Callback<void ()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:873:14 (libconsensus.so+0xea428) apache#18 kudu::internal::Invoker<1, kudu::internal::BindState<kudu::Callback<void ()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>, void ()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&), void ()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)>, void ()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>::Run(kudu::internal::BindStateBase*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:1065:12 (libconsensus.so+0xea3c3) apache#19 kudu::Callback<void ()()>::Run() const /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/callback.h:396:12 (libconsensus.so+0xa6dfd) apache#20 kudu::ClosureRunnable::Run() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:76:9 (libkudu_util.so+0x1cc9ad) apache#21 kudu::ThreadPool::DispatchThread() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:686:22 (libkudu_util.so+0x1c86d8) apache#22 boost::_mfi::mf0<void, kudu::ThreadPool>::operator()(kudu::ThreadPool*) const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkudu_util.so+0x1d3649) apache#23 void boost::_bi::list1<boost::_bi::value<kudu::ThreadPool*> >::operator()<boost::_mfi::mf0<void, kudu::ThreadPool>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf0<void, kudu::ThreadPool>&, boost::_bi::list0&, int) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkudu_util.so+0x1d359a) apache#24 boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::ThreadPool>, boost::_bi::list1<boost::_bi::value<kudu::ThreadPool*> > >::operator()() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkudu_util.so+0x1d3523) apache#25 boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::ThreadPool>, boost::_bi::list1<boost::_bi::value<kudu::ThreadPool*> > >, void>::invoke(boost::detail::function::function_buffer&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkudu_util.so+0x1d3319) apache#26 boost::function0<void>::operator()() const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xb6651) apache#27 kudu::Thread::SuperviseThread(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:615:3 (libkudu_util.so+0x1bfe34) Thread T59 'init [worker]-1' (tid=17081, running) created by main thread at: #0 pthread_create /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:992 (kudu-master+0x45af0b) apache#1 kudu::Thread::StartThread(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, boost::function<void ()()> const&, unsigned long, scoped_refptr<kudu::Thread>*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:559:15 (libkudu_util.so+0x1bf61b) apache#2 kudu::Status kudu::Thread::Create<void (kudu::ThreadPool::*)(), kudu::ThreadPool*>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, void (kudu::ThreadPool::* const&)(), kudu::ThreadPool* const&, scoped_refptr<kudu::Thread>*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.h:164:12 (libkudu_util.so+0x1ca9f5) apache#3 kudu::ThreadPool::CreateThread() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:749:10 (libkudu_util.so+0x1c7ce2) apache#4 kudu::ThreadPool::DoSubmit(std::__1::shared_ptr<kudu::Runnable>, kudu::ThreadPoolToken*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:556:21 (libkudu_util.so+0x1c64af) apache#5 kudu::ThreadPool::Submit(std::__1::shared_ptr<kudu::Runnable>) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:458:10 (libkudu_util.so+0x1c7f4f) apache#6 kudu::ThreadPool::SubmitClosure(kudu::Callback<void ()()>) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:450:10 (libkudu_util.so+0x1c7e91) apache#7 kudu::master::Master::StartAsync() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/master.cc:196:3 (libmaster.so+0x11f260) apache#8 kudu::master::Master::Start() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/master.cc:170:3 (libmaster.so+0x11ef25) apache#9 kudu::master::MasterMain(int, char**) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/master_main.cc:84:3 (kudu-master+0x4cb584) apache#10 main /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/master_main.cc:98:10 (kudu-master+0x4cb1be) Change-Id: I090a832b7fb25d8cb1e770c025048f73ac997eac Reviewed-on: http://gerrit.cloudera.org:8080/11818 Tested-by: Kudu Jenkins Reviewed-by: Alexey Serbin <[email protected]> Reviewed-by: Hao Hao <[email protected]>

The Messenger's lock is only intended to protect closing_, acceptor_pools_, and rpc_services_. This change adjusts its usage to reflect that: 1. There's no need to take the lock in the destructor. 2. It was held for longer than necessary in QueueInboundCall. 3. It wasn't needed at all in DumpConnections. The motivation for this was a TSAN lock inversion warning I saw in a precommit job, between the Messenger lock and glog's vmodule lock. The warning seems wrong (the vmodule lock is released after a VLOG statement ends), but one way to avoid it altogether is to not take the Messenger lock in its destructor. WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=5867) Cycle in lock order graph: M1870 (0x7b14000172f8) => M37857528269694952 (0x000000000000) => M1870 Mutex M37857528269694952 acquired here while holding mutex M1870 in main thread: #0 pthread_rwlock_wrlock /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1352 (kudu+0x4a360f) #1 glog_internal_namespace_::Mutex::Lock() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/glog-0.3.5/src/base/mutex.h:250:30 (libglog.so.0+0x1abe7) #2 glog_internal_namespace_::MutexLock::MutexLock(glog_internal_namespace_::Mutex*) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/glog-0.3.5/src/base/mutex.h:290 (libglog.so.0+0x1abe7) #3 google::InitVLOG3__(int**, int*, char const*, int) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/glog-0.3.5/src/vlog_is_on.cc:199 (libglog.so.0+0x1abe7) #4 kudu::rpc::Messenger::ShutdownInternal(kudu::rpc::Messenger::ShutdownMode) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/rpc/messenger.cc:283:5 (libkrpc.so+0xab101) #5 kudu::rpc::Messenger::AllExternalReferencesDropped() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/rpc/messenger.cc:249:3 (libkrpc.so+0xaaeb7) #6 std::__1::mem_fun_t<void, kudu::rpc::Messenger>::operator()(kudu::rpc::Messenger*) const /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/c++/v1/functional:1120:17 (libkrpc.so+0xaf3a5) #7 std::__1::__shared_ptr_pointer<kudu::rpc::Messenger*, std::__1::mem_fun_t<void, kudu::rpc::Messenger>, std::__1::allocator<kudu::rpc::Messenger> >::__on_zero_shared() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/c++/v1/memory:3586 (libkrpc.so+0xaf3a5) #8 std::__1::__shared_count::__release_shared() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/c++/v1/memory:3490:9 (kudu+0x56affe) #9 std::__1::__shared_weak_count::__release_shared() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/c++/v1/memory:3532 (kudu+0x56affe) #10 std::__1::shared_ptr<kudu::rpc::Messenger>::~shared_ptr() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/c++/v1/memory:4468 (kudu+0x56affe) #11 kudu::client::KuduClient::Data::~Data() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/client/client-internal.cc:179:1 (libkudu_client.so+0x136260) #12 kudu::client::KuduClient::~KuduClient() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/client/client.cc:394:3 (libkudu_client.so+0x1130cc) #13 std::__1::default_delete<kudu::client::KuduClient>::operator()(kudu::client::KuduClient*) const /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/c++/v1/memory:2285:5 (libkudu_client.so+0x12be1b) #14 std::__1::__shared_ptr_pointer<kudu::client::KuduClient*, std::__1::default_delete<kudu::client::KuduClient>, std::__1::allocator<kudu::client::KuduClient> >::__on_zero_shared() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/c++/v1/memory:3586 (libkudu_client.so+0x12be1b) #15 std::__1::__shared_count::__release_shared() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/c++/v1/memory:3490:9 (kudu+0x550d1e) #16 std::__1::__shared_weak_count::__release_shared() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/c++/v1/memory:3532 (kudu+0x550d1e) #17 std::__1::shared_ptr<kudu::client::KuduClient>::~shared_ptr() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/c++/v1/memory:4468 (kudu+0x550d1e) #18 kudu::tools::LeaderMasterProxy::~LeaderMasterProxy() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/tools/tool_action_common.h:233:7 (kudu+0x576cf9) #19 kudu::tools::(anonymous namespace)::ListMasters(kudu::tools::RunnerContext const&) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/tools/tool_action_master.cc:180:1 (kudu+0x572d0b) #20 _ZNSt3__18__invokeIRPFN4kudu6StatusERKNS1_5tools13RunnerContextEEJS6_EEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSA_DpOSB_ /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/c++/v1/type_traits:4482:1 (kudu+0x52e48b) #21 kudu::Status std::__1::__invoke_void_return_wrapper<kudu::Status>::__call<kudu::Status (*&)(kudu::tools::RunnerContext const&), kudu::tools::RunnerContext const&>(kudu::Status (*&)(kudu::tools::RunnerContext const&), kudu::tools::RunnerContext const&) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/c++/v1/__functional_base:318 (kudu+0x52e48b) #22 std::__1::__function::__func<kudu::Status (*)(kudu::tools::RunnerContext const&), std::__1::allocator<kudu::Status (*)(kudu::tools::RunnerContext const&)>, kudu::Status (kudu::tools::RunnerContext const&)>::operator()(kudu::tools::RunnerContext const&) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/c++/v1/functional:1562:12 (kudu+0x52e3bd) #23 std::__1::function<kudu::Status (kudu::tools::RunnerContext const&)>::operator()(kudu::tools::RunnerContext const&) const /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/c++/v1/functional:1916:12 (libkudu_tools_util.so+0x6c1c4) #24 kudu::tools::Action::Run(std::__1::vector<kudu::tools::Mode*, std::__1::allocator<kudu::tools::Mode*> > const&, std::__1::unordered_map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) const /home/jenkins-slave/workspace/kudu-master/2/src/kudu/tools/tool_action.cc:258:10 (libkudu_tools_util.so+0x6a8d4) #25 kudu::tools::DispatchCommand(std::__1::vector<kudu::tools::Mode*, std::__1::allocator<kudu::tools::Mode*> > const&, kudu::tools::Action*, std::__1::deque<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/tools/tool_main.cc:132:15 (kudu+0x5b42b6) #26 kudu::tools::RunTool(int, char**, bool) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/tools/tool_main.cc:204:16 (kudu+0x5b5211) #27 main /home/jenkins-slave/workspace/kudu-master/2/src/kudu/tools/tool_main.cc:265:10 (kudu+0x5b557e) Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message Mutex M1870 acquired here while holding mutex M37857528269694952 in thread T8: #0 AnnotateRWLockAcquired /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interface_ann.cc:271 (kudu+0x4d53ff) #1 kudu::rw_spinlock::lock() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/locks.h:112:5 (libkudu_client.so+0x177762) #2 kudu::percpu_rwlock::lock() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/locks.h:222:22 (libkudu_client.so+0x1776f2) #3 std::__1::lock_guard<kudu::percpu_rwlock>::lock_guard(kudu::percpu_rwlock&) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/c++/v1/__mutex_base:104:27 (libkrpc.so+0xac9c9) #4 kudu::rpc::Messenger::~Messenger() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/rpc/messenger.cc:430 (libkrpc.so+0xac9c9) #5 std::__1::default_delete<kudu::rpc::Messenger>::operator()(kudu::rpc::Messenger*) const /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/c++/v1/memory:2285:5 (libkrpc.so+0xb246b) #6 std::__1::__shared_ptr_pointer<kudu::rpc::Messenger*, std::__1::default_delete<kudu::rpc::Messenger>, std::__1::allocator<kudu::rpc::Messenger> >::__on_zero_shared() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/c++/v1/memory:3586 (libkrpc.so+0xb246b) #7 std::__1::__shared_count::__release_shared() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/c++/v1/memory:3490:9 (kudu+0x56affe) #8 std::__1::__shared_weak_count::__release_shared() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/c++/v1/memory:3532 (kudu+0x56affe) #9 std::__1::shared_ptr<kudu::rpc::Messenger>::~shared_ptr() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/c++/v1/memory:4468 (kudu+0x56affe) #10 std::__1::shared_ptr<kudu::rpc::Messenger>::reset() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/c++/v1/memory:4603:5 (libkrpc.so+0xc0771) #11 kudu::rpc::ReactorThread::RunThread() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/rpc/reactor.cc:499 (libkrpc.so+0xc0771) #12 boost::_mfi::mf0<void, kudu::rpc::ReactorThread>::operator()(kudu::rpc::ReactorThread*) const /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkrpc.so+0xca669) #13 void boost::_bi::list1<boost::_bi::value<kudu::rpc::ReactorThread*> >::operator()<boost::_mfi::mf0<void, kudu::rpc::ReactorThread>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf0<void, kudu::rpc::ReactorThread>&, boost::_bi::list0&, int) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkrpc.so+0xca5ba) #14 boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::rpc::ReactorThread>, boost::_bi::list1<boost::_bi::value<kudu::rpc::ReactorThread*> > >::operator()() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkrpc.so+0xca543) #15 boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::rpc::ReactorThread>, boost::_bi::list1<boost::_bi::value<kudu::rpc::ReactorThread*> > >, void>::invoke(boost::detail::function::function_buffer&) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkrpc.so+0xca339) #16 boost::function0<void>::operator()() const /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xba0b1) #17 kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/thread.cc:657:3 (libkudu_util.so+0x1ee174) Thread T8 'rpc reactor-588' (tid=5886, running) created by main thread at: #0 pthread_create /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:992 (kudu+0x490e36) #1 kudu::Thread::StartThread(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, boost::function<void ()> const&, unsigned long, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/thread.cc:601:15 (libkudu_util.so+0x1ed95b) #2 kudu::Status kudu::Thread::Create<void (kudu::rpc::ReactorThread::*)(), kudu::rpc::ReactorThread*>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, void (kudu::rpc::ReactorThread::* const&)(), kudu::rpc::ReactorThread* const&, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/thread.h:164:12 (libkrpc.so+0xc5a15) #3 kudu::rpc::ReactorThread::Init() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/rpc/reactor.cc:185:10 (libkrpc.so+0xc026e) #4 kudu::rpc::Reactor::Init() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/rpc/reactor.cc:759:18 (libkrpc.so+0xc4911) #5 kudu::rpc::Messenger::Init() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/rpc/messenger.cc:446:5 (libkrpc.so+0xaad72) #6 kudu::rpc::MessengerBuilder::Build(std::__1::shared_ptr<kudu::rpc::Messenger>*) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/rpc/messenger.cc:205:3 (libkrpc.so+0xaa7cd) #7 kudu::client::KuduClientBuilder::Build(std::__1::shared_ptr<kudu::client::KuduClient>*) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/client/client.cc:349:3 (libkudu_client.so+0x112561) #8 kudu::tools::LeaderMasterProxy::Init(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, kudu::MonoDelta const&) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/tools/tool_action_common.cc:786:30 (libkudu_tools_util.so+0x7740c) #9 kudu::tools::LeaderMasterProxy::Init(kudu::tools::RunnerContext const&) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/tools/tool_action_common.cc:792:10 (libkudu_tools_util.so+0x774d6) #10 kudu::tools::(anonymous namespace)::ListMasters(kudu::tools::RunnerContext const&) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/tools/tool_action_master.cc:109:3 (kudu+0x572be3) #11 _ZNSt3__18__invokeIRPFN4kudu6StatusERKNS1_5tools13RunnerContextEEJS6_EEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSA_DpOSB_ /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/c++/v1/type_traits:4482:1 (kudu+0x52e48b) #12 kudu::Status std::__1::__invoke_void_return_wrapper<kudu::Status>::__call<kudu::Status (*&)(kudu::tools::RunnerContext const&), kudu::tools::RunnerContext const&>(kudu::Status (*&)(kudu::tools::RunnerContext const&), kudu::tools::RunnerContext const&) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/c++/v1/__functional_base:318 (kudu+0x52e48b) #13 std::__1::__function::__func<kudu::Status (*)(kudu::tools::RunnerContext const&), std::__1::allocator<kudu::Status (*)(kudu::tools::RunnerContext const&)>, kudu::Status (kudu::tools::RunnerContext const&)>::operator()(kudu::tools::RunnerContext const&) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/c++/v1/functional:1562:12 (kudu+0x52e3bd) #14 std::__1::function<kudu::Status (kudu::tools::RunnerContext const&)>::operator()(kudu::tools::RunnerContext const&) const /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/c++/v1/functional:1916:12 (libkudu_tools_util.so+0x6c1c4) #15 kudu::tools::Action::Run(std::__1::vector<kudu::tools::Mode*, std::__1::allocator<kudu::tools::Mode*> > const&, std::__1::unordered_map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) const /home/jenkins-slave/workspace/kudu-master/2/src/kudu/tools/tool_action.cc:258:10 (libkudu_tools_util.so+0x6a8d4) #16 kudu::tools::DispatchCommand(std::__1::vector<kudu::tools::Mode*, std::__1::allocator<kudu::tools::Mode*> > const&, kudu::tools::Action*, std::__1::deque<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/tools/tool_main.cc:132:15 (kudu+0x5b42b6) #17 kudu::tools::RunTool(int, char**, bool) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/tools/tool_main.cc:204:16 (kudu+0x5b5211) #18 main /home/jenkins-slave/workspace/kudu-master/2/src/kudu/tools/tool_main.cc:265:10 (kudu+0x5b557e) Change-Id: I1fd93c06b14bc97a9ac4a37a5b6ca55ffa38f544 Reviewed-on: http://gerrit.cloudera.org:8080/14250 Tested-by: Kudu Jenkins Reviewed-by: Andrew Wong <[email protected]> Reviewed-by: Alexey Serbin <[email protected]>

The KernelStackWatchdog thread runs independently of the test thread, and by calling IsBeingDebugged, it winds up creating a trace event of its own. This is problematic given that trace-test sets up event callbacks to write to test fixture members, which go out of scope in between tests. The only solution I could find was to avoid starting the KernelStackWatchdog in trace-test by passing Thread::NO_STACK_WATCHDOG into thread creation. I also had to do this when creating the trace sampling thread, but given that's not on by default, I don't think it's so bad that we lose watchdog monitoring for it. To test, I ran trace-test under gdb and set a breakpoint in KernelStackWatchdog::RunThread. With the fix, gdb no longer hit that breakpoint. WARNING: ThreadSanitizer: data race (pid=4111) Read of size 8 at 0x0000015ba5c8 by thread T2: #0 kudu::TraceEventCallbackTest::Callback(long, char, unsigned char const*, char const*, unsigned long, int, char const* const*, unsigned char const*, unsigned long const*, unsigned char) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/trace-test.cc:463:5 (trace-test+0x4f107f) #1 kudu::debug::TraceLog::AddTraceEventWithThreadIdAndTimestamp(char, unsigned char const*, char const*, unsigned long, int, long const&, int, char const**, unsigned char const*, unsigned long const*, scoped_refptr<kudu::debug::ConvertableToTraceFormat> const*, unsigned char) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/debug/trace_event_impl.cc:1911:7 (libkudu_util.so+0x1208b3) #2 kudu::debug::TraceEventHandle trace_event_internal::AddTraceEventWithThreadIdAndTimestamp<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(char, unsigned char const*, char const*, unsigned long, int, long const&, unsigned char, char const*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/debug/trace_event.h:1314:10 (libkudu_util.so+0x146f58) #3 kudu::debug::TraceEventHandle trace_event_internal::AddTraceEvent<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(char, unsigned char const*, char const*, unsigned long, unsigned char, char const*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/debug/trace_event.h:1330:10 (libkudu_util.so+0x146bef) #4 kudu::(anonymous namespace)::PosixEnv::NewSequentialFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::unique_ptr<kudu::SequentialFile, std::__1::default_delete<kudu::SequentialFile> >*) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/env_posix.cc:1077:5 (libkudu_util.so+0x140905) #5 kudu::ReadFileToString(kudu::Env*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, kudu::faststring*) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/env.cc:73:19 (libkudu_util.so+0x140054) #6 kudu::IsBeingDebugged() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/os-util.cc:154:14 (libkudu_util.so+0x1c9687) #7 kudu::KernelStackWatchdog::RunThread() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/kernel_stack_watchdog.cc:141:9 (libkudu_util.so+0x17de59) #8 boost::_mfi::mf0<void, kudu::KernelStackWatchdog>::operator()(kudu::KernelStackWatchdog*) const /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkudu_util.so+0x17fd89) #9 void boost::_bi::list1<boost::_bi::value<kudu::KernelStackWatchdog*> >::operator()<boost::_mfi::mf0<void, kudu::KernelStackWatchdog>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf0<void, kudu::KernelStackWatchdog>&, boost::_bi::list0&, int) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkudu_util.so+0x17fcda) #10 boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::KernelStackWatchdog>, boost::_bi::list1<boost::_bi::value<kudu::KernelStackWatchdog*> > >::operator()() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkudu_util.so+0x17fc63) #11 boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::KernelStackWatchdog>, boost::_bi::list1<boost::_bi::value<kudu::KernelStackWatchdog*> > >, void>::invoke(boost::detail::function::function_buffer&) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkudu_util.so+0x17fa59) #12 boost::function0<void>::operator()() const /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkudu_util.so+0x1f1dd1) #13 kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/thread.cc:657:3 (libkudu_util.so+0x1ef3f4) Previous write of size 8 at 0x0000015ba5c8 by main thread: #0 kudu::TraceEventCallbackTest::SetUp() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/trace-test.cc:340:16 (trace-test+0x4f3a17) #1 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x552ef) #2 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x552ef) #3 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2470:3 (libgmock.so+0x343c1) #4 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x3574c) #5 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x36226) #6 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x425fa) #7 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x5625f) #8 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x5625f) #9 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x41ee2) #10 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x351b) #11 main /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/test_main.cc:106:13 (libkudu_test_main.so+0x2cc6) Location is global 'kudu::TraceEventCallbackTest::s_instance' of size 8 at 0x0000015ba5c8 (trace-test+0x0000015ba5c8) Thread T2 'kernel-watcher-' (tid=4116, running) created by main thread at: #0 pthread_create /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:992 (trace-test+0x453c86) #1 kudu::Thread::StartThread(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, boost::function<void ()> const&, unsigned long, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/thread.cc:601:15 (libkudu_util.so+0x1eebdb) #2 kudu::Status kudu::Thread::CreateWithFlags<boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::KernelStackWatchdog>, boost::_bi::list1<boost::_bi::value<kudu::KernelStackWatchdog*> > > >(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::KernelStackWatchdog>, boost::_bi::list1<boost::_bi::value<kudu::KernelStackWatchdog*> > > const&, unsigned long, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/thread.h:152:12 (libkudu_util.so+0x17eed1) #3 kudu::KernelStackWatchdog::KernelStackWatchdog() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/kernel_stack_watchdog.cc:71:3 (libkudu_util.so+0x17dc36) #4 Singleton<kudu::KernelStackWatchdog>::CreateInstance() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/gutil/singleton.h:124:18 (libkudu_util.so+0x17f664) #5 Singleton<kudu::KernelStackWatchdog>::Init() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/gutil/singleton.h:117:17 (libkudu_util.so+0x17f604) #6 GoogleOnceInternalInit(int*, void (*)(), void (*)(void*), void*) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/gutil/once.cc:43:7 (libgutil.so+0x2d7b3) #7 GoogleOnceInit(GoogleOnceType*, void (*)()) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/gutil/once.h:53:5 (libkudu_util.so+0x113e4d) #8 Singleton<kudu::KernelStackWatchdog>::get() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/gutil/singleton.h:79:5 (libkudu_util.so+0x17f5b1) #9 kudu::KernelStackWatchdog::GetInstance() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/kernel_stack_watchdog.h:87:12 (libkudu_util.so+0x17f423) #10 kudu::KernelStackWatchdog::CreateAndRegisterTLS() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/kernel_stack_watchdog.cc:219:3 (libkudu_util.so+0x17ed17) #11 kudu::KernelStackWatchdog::GetTLS() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/kernel_stack_watchdog.h:170:7 (libkudu_util.so+0x1f2901) #12 kudu::ScopedWatchKernelStack::ScopedWatchKernelStack(char const*, int) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/kernel_stack_watchdog.h:248:13 (libkudu_util.so+0x1f1b70) #13 kudu::Thread::StartThread(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, boost::function<void ()> const&, unsigned long, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/thread.cc:600:5 (libkudu_util.so+0x1eebaf) #14 kudu::Status kudu::Thread::Create<void (*)(int, int), int, int>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, void (* const&)(int, int), int const&, int const&, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/thread.h:170:12 (trace-test+0x4f03ef) #15 kudu::TraceTest_TestChromeTracing_Test::TestBody() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/trace-test.cc:172:5 (trace-test+0x4e750b) #16 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x552ef) #17 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x552ef) #18 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2474:5 (libgmock.so+0x344b8) #19 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x3574c) #20 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x36226) #21 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x425fa) #22 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x5625f) #23 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x5625f) #24 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x41ee2) #25 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x351b) #26 main /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/test_main.cc:106:13 (libkudu_test_main.so+0x2cc6) Change-Id: I5dc974be22ff101dcb8091be1fe692ab61376bc2 SUMMARY: ThreadSanitizer: data race /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/trace-test.cc:463:5 in kudu::TraceEventCallbackTest::Callback(long, char, unsigned char const*, char const*, unsigned long, int, char const* const*, unsigned char const*, unsigned long const*, unsigned char) Reviewed-on: http://gerrit.cloudera.org:8080/14256 Reviewed-by: Alexey Serbin <[email protected]> Tested-by: Adar Dembo <[email protected]>

Previously the FunctionGaugeDetacher in a Tablet could be destructed after some of the state in the Tablet had already been destructed. Specifically, the metrics would be detached after destructing 'last_rw_time_lock_' et al. This led to some TSAN warnings[1]. Without this patch, RaftConsensusNumLeadersMetricTest TestNumLeadersMetric, which deletes tablets and then immediately fetches metrics, would fail 29/1000 times in TSAN mode. With it, it only failed 7/1000, those due to an unrelated timeout (not addressed in this patch). I went ahead and updated other misordered instances of FunctionGaugeDetachers. As far as I can tell, there aren't any other affected tests. [1]: ================== WARNING: ThreadSanitizer: data race (pid=30286) Write of size 1 at 0x7b58000521c8 by thread T156 (mutexes: write M3961, write M2995): #0 AnnotateRWLockDestroy /data/8/awong/kudu/thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interface_ann.cc:264 (kudu-tserver+0x4ab83e) #1 kudu::rw_spinlock::~rw_spinlock() ../src/kudu/util/locks.h:89:5 (libtserver.so+0x262d4b) #2 kudu::tablet::Tablet::~Tablet() ../src/kudu/tablet/tablet.cc:270:1 (libtablet.so+0x174886) #3 std::__1::default_delete<kudu::tablet::Tablet>::operator()(kudu::tablet::Tablet*) const /data/8/awong/kudu/thirdparty/installed/tsan/include/c++/v1/memory:2338:5 (libtablet.so+0x233026) #4 std::__1::__shared_ptr_pointer<kudu::tablet::Tablet*, std::__1::default_delete<kudu::tablet::Tablet>, std::__1::allocator<kudu::tablet::Tablet> >::__on_zero_shared() /data/8/awong/kudu/thirdparty/installed/tsan/include/c++/v1/memory:3511:5 (libtablet.so+0x2343bf) #5 std::__1::__shared_count::__release_shared() /data/8/awong/kudu/thirdparty/installed/tsan/include/c++/v1/memory:3415:9 (libtserver.so+0x125c9c) #6 std::__1::__shared_weak_count::__release_shared() /data/8/awong/kudu/thirdparty/installed/tsan/include/c++/v1/memory:3457:27 (libtserver.so+0x125c12) #7 std::__1::shared_ptr<kudu::tablet::Tablet>::~shared_ptr() /data/8/awong/kudu/thirdparty/installed/tsan/include/c++/v1/memory:4393:19 (libtserver.so+0x1478b7) #8 std::__1::shared_ptr<kudu::tablet::Tablet>::reset() /data/8/awong/kudu/thirdparty/installed/tsan/include/c++/v1/memory:4528:5 (libtablet.so+0x2622c6) #9 kudu::tablet::TabletReplica::Stop() ../src/kudu/tablet/tablet_replica.cc:318:13 (libtablet.so+0x2578bd) #10 kudu::tserver::TSTabletManager::DeleteTablet(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, kudu::tablet::TabletDataState, boost::optional<long> const&, kudu::tserver::TabletServerErrorPB_Code*) ../src/kudu/tserver/ts_tablet_manager.cc:972:12 (libtserver.so+0x25a568) #11 kudu::tserver::DeleteTabletRunnable::Run() ../src/kudu/tserver/ts_tablet_manager.cc:882:36 (libtserver.so+0x27caa8) #12 kudu::ThreadPool::DispatchThread() ../src/kudu/util/threadpool.cc:685:22 (libkudu_util.so+0x40e2f6) #13 boost::_mfi::mf0<void, kudu::ThreadPool>::operator()(kudu::ThreadPool*) const ../thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkudu_util.so+0x4357bc) #14 void boost::_bi::list1<boost::_bi::value<kudu::ThreadPool*> >::operator()<boost::_mfi::mf0<void, kudu::ThreadPool>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf0<void, kudu::ThreadPool>&, boost::_bi::list0&, int) ../thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkudu_util.so+0x43566d) #15 boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::ThreadPool>, boost::_bi::list1<boost::_bi::value<kudu::ThreadPool*> > >::operator()() ../thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkudu_util.so+0x4355b1) #16 boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::ThreadPool>, boost::_bi::list1<boost::_bi::value<kudu::ThreadPool*> > >, void>::invoke(boost::detail::function::function_buffer&) ../thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkudu_util.so+0x4351e0) #17 boost::function0<void>::operator()() const ../thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0x13b9c1) #18 kudu::Thread::SuperviseThread(void*) ../src/kudu/util/thread.cc:675:3 (libkudu_util.so+0x3eed69) Previous atomic write of size 4 at 0x7b58000521c8 by thread T12 (mutexes: write M261907054171829296): #0 __tsan_atomic32_compare_exchange_strong /data/8/awong/kudu/thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cc:780 (kudu-tserver+0x4b475e) #1 base::subtle::Release_CompareAndSwap(int volatile*, int, int) ../src/kudu/gutil/atomicops-internals-tsan.h:93:3 (libtablet.so+0x1d4842) #2 kudu::rw_semaphore::unlock_shared() ../src/kudu/util/rw_semaphore.h:91:19 (libtablet.so+0x1d4766) #3 kudu::rw_spinlock::unlock_shared() ../src/kudu/util/locks.h:99:10 (libtablet.so+0x1d45da) #4 kudu::shared_lock<kudu::rw_spinlock>::~shared_lock() ../src/kudu/util/locks.h:283:11 (libtablet.so+0x1a3bf5) #5 kudu::tablet::Tablet::LastReadElapsedSeconds() const ../src/kudu/tablet/tablet.cc:1989:1 (libtablet.so+0x17457c) #6 kudu::internal::RunnableAdapter<unsigned long (kudu::tablet::Tablet::*)() const>::Run(kudu::tablet::Tablet const*) ../src/kudu/gutil/bind_internal.h:155:12 (libtablet.so+0x1e5c1c) #7 kudu::internal::InvokeHelper<false, unsigned long, kudu::internal::RunnableAdapter<unsigned long (kudu::tablet::Tablet::*)() const>, void (kudu::tablet::Tablet*)>::MakeItSo(kudu::internal::RunnableAdapter<unsigned long (kudu::tablet::Tablet::*)() const>, kudu::tablet::Tablet*) ../src/kudu/gutil/bind_internal.h:865:21 (libtablet.so+0x1e5a8c) #8 kudu::internal::Invoker<1, kudu::internal::BindState<kudu::internal::RunnableAdapter<unsigned long (kudu::tablet::Tablet::*)() const>, unsigned long (kudu::tablet::Tablet const*), void (kudu::internal::UnretainedWrapper<kudu::tablet::Tablet>)>, unsigned long (kudu::tablet::Tablet const*)>::Run(kudu::internal::BindStateBase*) ../src/kudu/gutil/bind_internal.h:1065:12 (libtablet.so+0x1e59a7) #9 kudu::Callback<unsigned long ()>::Run() const ../src/kudu/gutil/callback.h:396:12 (libtserver.so+0x15acfb) #10 kudu::FunctionGauge<unsigned long>::value() const ../src/kudu/util/metrics.h:1239:22 (libtserver.so+0x15ab1f) #11 kudu::FunctionGauge<unsigned long>::WriteValue(kudu::JsonWriter*) const ../src/kudu/util/metrics.h:1243:19 (libtserver.so+0x15a7bc) #12 kudu::Gauge::WriteAsJson(kudu::JsonWriter*, kudu::MetricJsonOptions const&) const ../src/kudu/util/metrics.cc:716:3 (libkudu_util.so+0x31da17) #13 void kudu::WriteMetricsToJson<std::__1::unordered_map<kudu::MetricPrototype const*, scoped_refptr<kudu::Metric>, std::__1::hash<kudu::MetricPrototype const*>, std::__1::equal_to<kudu::MetricPrototype const*>, std::__1::allocator<std::__1::pair<kudu::MetricPrototype const* const, scoped_refptr<kudu::Metric> > > > >(kudu::JsonWriter*, std::__1::unordered_map<kudu::MetricPrototype const*, scoped_refptr<kudu::Metric>, std::__1::hash<kudu::MetricPrototype const*>, std::__1::equal_to<kudu::MetricPrototype const*>, std::__1::allocator<std::__1::pair<kudu::MetricPrototype const* const, scoped_refptr<kudu::Metric> > > > const&, kudu::MetricJsonOptions const&) ../src/kudu/util/metrics.cc:64:7 (libkudu_util.so+0x321ff2) #14 kudu::MetricEntity::WriteAsJson(kudu::JsonWriter*, kudu::MetricJsonOptions const&) const ../src/kudu/util/metrics.cc:388:3 (libkudu_util.so+0x31a9f4) #15 kudu::MetricRegistry::WriteAsJson(kudu::JsonWriter*, kudu::MetricJsonOptions const&) const ../src/kudu/util/metrics.cc:517:7 (libkudu_util.so+0x31c34a) #16 kudu::server::DiagnosticsLog::LogMetrics() ../src/kudu/server/diagnostics_log.cc:345:3 (libserver_process.so+0xac51a) #17 kudu::server::DiagnosticsLog::RunThread() ../src/kudu/server/diagnostics_log.cc:225:7 (libserver_process.so+0xabc50) #18 boost::_mfi::mf0<void, kudu::server::DiagnosticsLog>::operator()(kudu::server::DiagnosticsLog*) const ../thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libserver_process.so+0xb88ac) #19 void boost::_bi::list1<boost::_bi::value<kudu::server::DiagnosticsLog*> >::operator()<boost::_mfi::mf0<void, kudu::server::DiagnosticsLog>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf0<void, kudu::server::DiagnosticsLog>&, boost::_bi::list0&, int) ../thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libserver_process.so+0xb875d) #20 boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::server::DiagnosticsLog>, boost::_bi::list1<boost::_bi::value<kudu::server::DiagnosticsLog*> > >::operator()() ../thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libserver_process.so+0xb8671) #21 boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::server::DiagnosticsLog>, boost::_bi::list1<boost::_bi::value<kudu::server::DiagnosticsLog*> > >, void>::invoke(boost::detail::function::function_buffer&) ../thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libserver_process.so+0xb82a0) #22 boost::function0<void>::operator()() const ../thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0x13b9c1) #23 kudu::Thread::SuperviseThread(void*) ../src/kudu/util/thread.cc:675:3 (libkudu_util.so+0x3eed69) Change-Id: Ib32120178a68b5389e167643e9bb8b89f8c625b9 Reviewed-on: http://gerrit.cloudera.org:8080/14979 Reviewed-by: Alexey Serbin <[email protected]> Tested-by: Kudu Jenkins

It previously possible for a CommitTask to be destructed before completing the loop of scheduling all asynchronous tasks. This led to a race as seen below: WARNING: ThreadSanitizer: data race (pid=32435) Write of size 8 at 0x7b1c000ce2d8 by thread T105 (mutexes: write M424881254664896540): #0 std::__1::__vector_base<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >::__destruct_at_end(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*) /data/3/awong/Repositories/kudu/thirdparty/installed/tsan/include/c++/v1/vector:427:12 (txn_commit-itest+0x576cb1) #1 std::__1::__vector_base<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >::clear() /data/3/awong/Repositories/kudu/thirdparty/installed/tsan/include/c++/v1/vector:369:29 (txn_commit-itest+0x5770d1) #2 std::__1::__vector_base<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >::~__vector_base() /data/3/awong/Repositories/kudu/thirdparty/installed/tsan/include/c++/v1/vector:463:9 (txn_commit-itest+0x59caf9) #3 std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >::~vector() /data/3/awong/Repositories/kudu/thirdparty/installed/tsan/include/c++/v1/vector:555:5 (libtransactions.so+0x8c2a0) #4 kudu::transactions::CommitTasks::~CommitTasks() ../src/kudu/transactions/txn_status_manager.h:177:26 (libtransactions.so+0xcce8b) #5 kudu::RefCountedThreadSafe<kudu::transactions::CommitTasks, kudu::DefaultRefCountedThreadSafeTraits<kudu::transactions::CommitTasks> >::DeleteInternal(kudu::transactions::CommitTasks const*) ../src/kudu/gutil/ref_counted.h:153:44 (libtransactions.so+0xcce1a) #6 kudu::DefaultRefCountedThreadSafeTraits<kudu::transactions::CommitTasks>::Destruct(kudu::transactions::CommitTasks const*) ../src/kudu/gutil/ref_counted.h:116:5 (libtransactions.so+0xccdc8) #7 kudu::RefCountedThreadSafe<kudu::transactions::CommitTasks, kudu::DefaultRefCountedThreadSafeTraits<kudu::transactions::CommitTasks> >::Release() const ../src/kudu/gutil/ref_counted.h:144:7 (libtransactions.so+0xccd70) #8 scoped_refptr<kudu::transactions::CommitTasks>::~scoped_refptr() ../src/kudu/gutil/ref_counted.h:266:13 (libtransactions.so+0xbf785) #9 std::__1::pair<long const, scoped_refptr<kudu::transactions::CommitTasks> >::~pair() /data/3/awong/Repositories/kudu/thirdparty/installed/tsan/include/c++/v1/utility:315:29 (libtransactions.so+0xc7652) #10 void std::__1::allocator_traits<std::__1::allocator<std::__1::__hash_node<std::__1::__hash_value_type<long, scoped_refptr<kudu::transactions::CommitTasks> >, void*> > >::__destroy<std::__1::pair<long const, scoped_refptr<kudu::transactions::CommitTasks> > >(std::__1::integral_constant<bool, false>, std::__1::allocator<std::__1::__hash_node<std::__1::__hash_value_type<long, scoped_refptr<kudu::transactions::CommitTasks> >, void*> >&, std::__1::pair<long const, scoped_refptr<kudu::transactions::CommitTasks> >*) /data/3/awong/Repositories/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1747:23 (libtransactions.so+0xc7614) #11 void std::__1::allocator_traits<std::__1::allocator<std::__1::__hash_node<std::__1::__hash_value_type<long, scoped_refptr<kudu::transactions::CommitTasks> >, void*> > >::destroy<std::__1::pair<long const, scoped_refptr<kudu::transactions::CommitTasks> > >(std::__1::allocator<std::__1::__hash_node<std::__1::__hash_value_type<long, scoped_refptr<kudu::transactions::CommitTasks> >, void*> >&, std::__1::pair<long const, scoped_refptr<kudu::transactions::CommitTasks> >*) /data/3/awong/Repositories/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1595:14 (libtransactions.so+0xc7518) #12 std::__1::__hash_node_destructor<std::__1::allocator<std::__1::__hash_node<std::__1::__hash_value_type<long, scoped_refptr<kudu::transactions::CommitTasks> >, void*> > >::operator()(std::__1::__hash_node<std::__1::__hash_value_type<long, scoped_refptr<kudu::transactions::CommitTasks> >, void*>*) /data/3/awong/Repositories/kudu/thirdparty/installed/tsan/include/c++/v1/__hash_table:844:13 (libtransactions.so+0xc740d) #13 std::__1::unique_ptr<std::__1::__hash_node<std::__1::__hash_value_type<long, scoped_refptr<kudu::transactions::CommitTasks> >, void*>, std::__1::__hash_node_destructor<std::__1::allocator<std::__1::__hash_node<std::__1::__hash_value_type<long, scoped_refptr<kudu::transactions::CommitTasks> >, void*> > > >::reset(std::__1::__hash_node<std::__1::__hash_value_type<long, scoped_refptr<kudu::transactions::CommitTasks> >, void*>*) /data/3/awong/Repositories/kudu/thirdparty/installed/tsan/include/c++/v1/memory:2593:7 (libtransactions.so+0xc72e0) #14 std::__1::unique_ptr<std::__1::__hash_node<std::__1::__hash_value_type<long, scoped_refptr<kudu::transactions::CommitTasks> >, void*>, std::__1::__hash_node_destructor<std::__1::allocator<std::__1::__hash_node<std::__1::__hash_value_type<long, scoped_refptr<kudu::transactions::CommitTasks> >, void*> > > >::~unique_ptr() /data/3/awong/Repositories/kudu/thirdparty/installed/tsan/include/c++/v1/memory:2547:19 (libtransactions.so+0xc6cbc) #15 std::__1::__hash_table<std::__1::__hash_value_type<long, scoped_refptr<kudu::transactions::CommitTasks> >, std::__1::__unordered_map_hasher<long, std::__1::__hash_value_type<long, scoped_refptr<kudu::transactions::CommitTasks> >, std::__1::hash<long>, true>, std::__1::__unordered_map_equal<long, std::__1::__hash_value_type<long, scoped_refptr<kudu::transactions::CommitTasks> >, std::__1::equal_to<long>, true>, std::__1::allocator<std::__1::__hash_value_type<long, scoped_refptr<kudu::transactions::CommitTasks> > > >::erase(std::__1::__hash_const_iterator<std::__1::__hash_node<std::__1::__hash_value_type<long, scoped_refptr<kudu::transactions::CommitTasks> >, void*>*>) /data/3/awong/Repositories/kudu/thirdparty/installed/tsan/include/c++/v1/__hash_table:2598:5 (libtransactions.so+0xc676e) #16 std::__1::unordered_map<long, scoped_refptr<kudu::transactions::CommitTasks>, std::__1::hash<long>, std::__1::equal_to<long>, std::__1::allocator<std::__1::pair<long const, scoped_refptr<kudu::transactions::CommitTasks> > > >::erase(std::__1::__hash_map_iterator<std::__1::__hash_iterator<std::__1::__hash_node<std::__1::__hash_value_type<long, scoped_refptr<kudu::transactions::CommitTasks> >, void*>*> >) /data/3/awong/Repositories/kudu/thirdparty/installed/tsan/include/c++/v1/unordered_map:1193:57 (libtransactions.so+0xc5b40) #17 kudu::transactions::TxnStatusManager::RemoveCommitTask(long, kudu::transactions::CommitTasks const*) ../src/kudu/transactions/txn_status_manager.h:433:26 (libtransactions.so+0xbefc6) #18 kudu::transactions::CommitTasks::IsShuttingDownCleanupIfLastOp() ../src/kudu/transactions/txn_status_manager.cc:181:28 (libtransactions.so+0x97dea) #19 kudu::transactions::CommitTasks::AbortTxnAsyncTask(int)::$_2::operator()(kudu::Status const&) const ../src/kudu/transactions/txn_status_manager.cc:319:9 (libtransactions.so+0xaefd6) #20 decltype(std::__1::forward<kudu::transactions::CommitTasks::AbortTxnAsyncTask(int)::$_2&>(fp)(std::__1::forward<kudu::Status const&>(fp0))) std::__1::__invoke<kudu::transactions::CommitTasks::AbortTxnAsyncTask(int)::$_2&, kudu::Status const&>(kudu::transactions::CommitTasks::AbortTxnAsyncTask(int)::$_2&, kudu::Status const&) /data/3/awong/Repositories/kudu/thirdparty/installed/tsan/include/c++/v1/type_traits:3530:1 (libtransactions.so+0xaeefd) #21 void std::__1::__invoke_void_return_wrapper<void>::__call<kudu::transactions::CommitTasks::AbortTxnAsyncTask(int)::$_2&, kudu::Status const&>(kudu::transactions::CommitTasks::AbortTxnAsyncTask(int)::$_2&, kudu::Status const&) /data/3/awong/Repositories/kudu/thirdparty/installed/tsan/include/c++/v1/__functional_base:348:9 (libtransactions.so+0xaee3d) #22 std::__1::__function::__alloc_func<kudu::transactions::CommitTasks::AbortTxnAsyncTask(int)::$_2, std::__1::allocator<kudu::transactions::CommitTasks::AbortTxnAsyncTask(int)::$_2>, void (kudu::Status const&)>::operator()(kudu::Status const&) /data/3/awong/Repositories/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1533:16 (libtransactions.so+0xaedbd) #23 std::__1::__function::__func<kudu::transactions::CommitTasks::AbortTxnAsyncTask(int)::$_2, std::__1::allocator<kudu::transactions::CommitTasks::AbortTxnAsyncTask(int)::$_2>, void (kudu::Status const&)>::operator()(kudu::Status const&) /data/3/awong/Repositories/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1707:12 (libtransactions.so+0xad06c) #24 std::__1::__function::__value_func<void (kudu::Status const&)>::operator()(kudu::Status const&) const /data/3/awong/Repositories/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1860:16 (libmaster.so+0x32ca24) #25 std::__1::function<void (kudu::Status const&)>::operator()(kudu::Status const&) const /data/3/awong/Repositories/kudu/thirdparty/installed/tsan/include/c++/v1/functional:2419:12 (libmaster.so+0x31d80b) #26 kudu::transactions::ParticipantRpc::Finish(kudu::Status const&) ../src/kudu/transactions/participant_rpc.cc:227:3 (libtransactions.so+0x7f3e7) ... Previous read of size 8 at 0x7b1c000ce2d8 by thread T186 (mutexes: read M322424363142217872): #0 std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >::size() const /data/3/awong/Repositories/kudu/thirdparty/installed/tsan/include/c++/v1/vector:656:46 (libtransactions.so+0x8d2f9) #1 kudu::transactions::CommitTasks::AbortTxnAsync() ../src/kudu/transactions/txn_status_manager.cc:365:42 (libtransactions.so+0x989d2) #2 kudu::transactions::TxnStatusManager::BeginAbortTransaction(long, boost::optional<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > const&, kudu::tserver::TabletServerErrorPB*) ../src/kudu/transactions/txn_status_manager.cc:1219:25 (libtransactions.so+0xa3cc6) #3 kudu::transactions::CommitTasks::ScheduleBeginAbortTxnWrite()::$_3::operator()() const ../src/kudu/transactions/txn_status_manager.cc:378:3 (libtransactions.so+0xb245d) #4 decltype(std::__1::forward<kudu::transactions::CommitTasks::ScheduleBeginAbortTxnWrite()::$_3&>(fp)()) std::__1::__invoke<kudu::transactions::CommitTasks::ScheduleBeginAbortTxnWrite()::$_3&>(kudu::transactions::CommitTasks::ScheduleBeginAbortTxnWrite()::$_3&) /data/3/awong/Repositories/kudu/thirdparty/installed/tsan/include/c++/v1/type_traits:3530:1 (libtransactions.so+0xb2180) #5 void std::__1::__invoke_void_return_wrapper<void>::__call<kudu::transactions::CommitTasks::ScheduleBeginAbortTxnWrite()::$_3&>(kudu::transactions::CommitTasks::ScheduleBeginAbortTxnWrite()::$_3&) /data/3/awong/Repositories/kudu/thirdparty/installed/tsan/include/c++/v1/__functional_base:348:9 (libtransactions.so+0xb20e0) #6 std::__1::__function::__alloc_func<kudu::transactions::CommitTasks::ScheduleBeginAbortTxnWrite()::$_3, std::__1::allocator<kudu::transactions::CommitTasks::ScheduleBeginAbortTxnWrite()::$_3>, void ()>::operator()() /data/3/awong/Repositories/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1533:16 (libtransactions.so+0xb2080) #7 std::__1::__function::__func<kudu::transactions::CommitTasks::ScheduleBeginAbortTxnWrite()::$_3, std::__1::allocator<kudu::transactions::CommitTasks::ScheduleBeginAbortTxnWrite()::$_3>, void ()>::operator()() /data/3/awong/Repositories/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1707:12 (libtransactions.so+0xb042f) #8 std::__1::__function::__value_func<void ()>::operator()() const /data/3/awong/Repositories/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1860:16 (libtserver_test_util.so+0x58396) #9 std::__1::function<void ()>::operator()() const /data/3/awong/Repositories/kudu/thirdparty/installed/tsan/include/c++/v1/functional:2419:12 (libtserver_test_util.so+0x58098) ... This patch fixes this by caching the size before iterating. Prior to this patch, the test failed in TSAN mode 3/100 times. With this patch, it passed 1000/1000 times. Change-Id: Ic974354b300f2a6c1b04505e740249273f33b80c Reviewed-on: http://gerrit.cloudera.org:8080/17283 Reviewed-by: Alexey Serbin <[email protected]> Tested-by: Kudu Jenkins

We recently added a few test cases where the client negotiation fails with this error (which is what we expect): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server kudu/[email protected] not found in Kerberos database) Apparently SASL doesn't allocate enough memory for this error message in some cases which causes these tests to be flaky with a ~20% error rate with AddressSanitizer enabled: ==9298==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60e00003e2d6 at pc 0x000000530bf4 bp 0x7f8eb50ad0f0 sp 0x7f8eb50ac8a0 READ of size 151 at 0x60e00003e2d6 thread T88 (client-negotiat) #0 0x530bf3 in __interceptor_strlen.part.35 sanitizer_common/sanitizer_common_interceptors.inc:365:5 #1 0x7f8ee6ad9ee8 in std::basic_ostream<char, std::char_traits<char> >& std::operator<<<std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*) (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0x113ee8) #2 0x7f8eeb7c9c9b in kudu::rpc::SaslLogCallback(void*, int, char const*) ../src/kudu/rpc/sasl_common.cc:102:29 #3 0x7f8eeb30241c in sasl_seterror (/tmp/dist-test-taskexUtyr/build/dist-test-system-libs/libsasl2.so.3+0x1441c) #4 0x7f8edd8f143d in _init (/tmp/dist-test-taskexUtyr/build/dist-test-system-libs/sasl2/libgssapiv2.so+0x243d) #5 0x7f8edd8f2452 in _init (/tmp/dist-test-taskexUtyr/build/dist-test-system-libs/sasl2/libgssapiv2.so+0x3452) #6 0x7f8eeb2f7844 in sasl_client_step (/tmp/dist-test-taskexUtyr/build/dist-test-system-libs/libsasl2.so.3+0x9844) #7 0x7f8eeb2f7bc5 in sasl_client_start (/tmp/dist-test-taskexUtyr/build/dist-test-system-libs/libsasl2.so.3+0x9bc5) #8 0x7f8eeb678679 in kudu::rpc::ClientNegotiation::SendSaslInitiate()::$_1::operator()() const ../src/kudu/rpc/client_negotiation.cc:594:14 #9 0x7f8eeb67831c in std::_Function_handler<int (), kudu::rpc::ClientNegotiation::SendSaslInitiate()::$_1>::_M_invoke(std::_Any_data const&) ../../../include/c++/8/bits/std_function.h:282:9 #10 0x7f8ef3b28220 in std::function<int ()>::operator()() const ../../../include/c++/8/bits/std_function.h:687:14 #11 0x7f8eeb7c5840 in kudu::rpc::WrapSaslCall(sasl_conn*, std::function<int ()> const&, char const*) ../src/kudu/rpc/sasl_common.cc:341:12 #12 0x7f8eeb67363b in kudu::rpc::ClientNegotiation::SendSaslInitiate() ../src/kudu/rpc/client_negotiation.cc:593:20 #13 0x7f8eeb66e0c7 in kudu::rpc::ClientNegotiation::AuthenticateBySasl(kudu::faststring*, std::unique_ptr<kudu::rpc::ErrorStatusPB, std::default_delete<kudu::rpc::ErrorStatusPB> >*) ../src/kudu/rpc/client_negotiation.cc:523:14 #14 0x7f8eeb667b99 in kudu::rpc::ClientNegotiation::Negotiate(std::unique_ptr<kudu::rpc::ErrorStatusPB, std::default_delete<kudu::rpc::ErrorStatusPB> >*) ../src/kudu/rpc/client_negotiation.cc:220:7 #15 0x7f8eeb715027 in kudu::rpc::DoClientNegotiation(kudu::rpc::Connection*, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime, std::unique_ptr<kudu::rpc::ErrorStatusPB, std::default_delete<kudu::rpc::ErrorStatusPB> >*) ../src/kudu/rpc/negotiation.cc:218:3 #16 0x7f8eeb712095 in kudu::rpc::Negotiation::RunNegotiation(scoped_refptr<kudu::rpc::Connection> const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime) ../src/kudu/rpc/negotiation.cc:295:9 #17 0x7f8eeb74d4ad in kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1::operator()() const ../src/kudu/rpc/reactor.cc:614:3 #18 0x7f8eeb74d06c in std::_Function_handler<void (), kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1>::_M_invoke(std::_Any_data const&) ../../../include/c++/8/bits/std_function.h:297:2 #19 0x71b760 in std::function<void ()>::operator()() const ../../../include/c++/8/bits/std_function.h:687:14 #20 0x7f8ee917d03d in kudu::ThreadPool::DispatchThread() ../src/kudu/util/threadpool.cc:669:7 #21 0x7f8ee91817dc in kudu::ThreadPool::CreateThread()::$_1::operator()() const ../src/kudu/util/threadpool.cc:742:48 #22 0x7f8ee918162c in std::_Function_handler<void (), kudu::ThreadPool::CreateThread()::$_1>::_M_invoke(std::_Any_data const&) ../../../include/c++/8/bits/std_function.h:297:2 #23 0x71b760 in std::function<void ()>::operator()() const ../../../include/c++/8/bits/std_function.h:687:14 #24 0x7f8ee915660a in kudu::Thread::SuperviseThread(void*) ../src/kudu/util/thread.cc:674:3 #25 0x7f8eec6106da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da) #26 0x7f8ee64de71e in clone (/lib/x86_64-linux-gnu/libc.so.6+0x12171e) 0x60e00003e2d6 is located 0 bytes to the right of 150-byte region [0x60e00003e240,0x60e00003e2d6) allocated by thread T88 (client-negotiat) here: #0 0x5a4bb8 in malloc /home/abukor/src/kudu/thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:145:3 #1 0x7f8eeb2fa1df in _buf_alloc (/tmp/dist-test-taskexUtyr/build/dist-test-system-libs/libsasl2.so.3+0xc1df) This patch suppresses address sanitizer errors in sasl_seterror(). Change-Id: Ie66e1f14c9750b13676c7e28e6439057a5e73341 Reviewed-on: http://gerrit.cloudera.org:8080/17317 Tested-by: Attila Bukor <[email protected]> Reviewed-by: Alexey Serbin <[email protected]> Reviewed-by: Grant Henke <[email protected]>

The thread pool of the DNS resolver should be shut down along with the messenger in ServerBase to prevent retrying of RPCs that failed as a collateral of the shutdown process in progress. Those RPCs might be retried by invoking rpc::Proxy::RefreshDnsAndEnqueueRequest(), etc. On the related note, I also added a guard to protect ThreadPool::tokens_ in the destructor of the ThreadPool class, as elsewhere. I also snuck in an update to call DCHECK() in a loop only when DCHECK_IS_ON() macro evaluates to 'true'. This addresses flakiness reported at least in one of the RemoteKsckTest scenarios (e.g., TestFilterOnNotabletTable in [1]). One of the related TSAN reports looked like below: RemoteKsckTest.TestFilterOnNotabletTable: WARNING: ThreadSanitizer: data race Read of size 8 at 0x7b54001e5118 by main thread: #0 std::__1::__hash_table<kudu::ThreadPoolToken*, ...>::size() const #1 std::__1::unordered_set<kudu::ThreadPoolToken*, ...>::size() const #2 kudu::ThreadPool::~ThreadPool() ... #6 kudu::kserver::KuduServer::~KuduServer() #7 kudu::tserver::TabletServer::~TabletServer() ... Previous write of size 8 at 0x7b54001e5118 by thread T262 ...: #0 std::__1::__hash_table<kudu::ThreadPoolToken*, ...>::remove(...) ... #4 kudu::ThreadPool::ReleaseToken(...) #5 kudu::ThreadPoolToken::~ThreadPoolToken() ... #24 kudu::consensus::LeaderElection::~LeaderElection() ... #35 kudu::rpc::Proxy::RefreshDnsAndEnqueueRequest(...) ... #41 kudu::DnsResolver::RefreshAddressesAsync() ... Thread T262 'dns-resolver [w' (tid=29102, running) created by thread T182 at: #0 pthread_create #1 kudu::Thread::StartThread(...) #2 kudu::Thread::Create(...) #3 kudu::ThreadPool::CreateThread() #4 kudu::ThreadPool::DoSubmit(..., kudu::ThreadPoolToken*) #5 kudu::ThreadPool::Submit(...) #6 kudu::DnsResolver::RefreshAddressesAsync(..) #7 kudu::rpc::Proxy::RefreshDnsAndEnqueueRequest(...) #8 kudu::rpc::Proxy::AsyncRequest(...) ... #15 kudu::rpc::OutboundCall::CallCallback() #16 kudu::rpc::OutboundCall::SetFailed() #17 kudu::rpc::Connection::Shutdown() #18 kudu::rpc::ReactorThread::ShutdownInternal() ... #25 kudu::rpc::ReactorThread::RunThread() ... [1] http://dist-test.cloudera.org:8080/test_drilldown?test_name=ksck_remote-test Change-Id: I525f1078a349dbd2926938bb4fcc3e80888dfbb4 Reviewed-on: http://gerrit.cloudera.org:8080/22434 Tested-by: Alexey Serbin <[email protected]> Reviewed-by: Abhishek Chennaka <[email protected]>

check session when applying operation

aa74af4

granthenke closed this Jan 18, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

check session when applying operation #15

check session when applying operation #15

KarlManong commented Jan 9, 2019

adembo commented Jan 9, 2019

KarlManong commented Jan 10, 2019

adembo commented Jan 11, 2019

KarlManong commented Jan 11, 2019

adembo commented Jan 18, 2019

granthenke commented Jan 18, 2019

check session when applying operation #15

check session when applying operation #15

Conversation

KarlManong commented Jan 9, 2019

adembo commented Jan 9, 2019

KarlManong commented Jan 10, 2019

adembo commented Jan 11, 2019

KarlManong commented Jan 11, 2019

adembo commented Jan 18, 2019

granthenke commented Jan 18, 2019