Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ASAN] heap-buffer-overflow in L1TrackerEtMissEmulatorProducer #41320

Open
makortel opened this issue Apr 11, 2023 · 16 comments
Open

[ASAN] heap-buffer-overflow in L1TrackerEtMissEmulatorProducer #41320

makortel opened this issue Apr 11, 2023 · 16 comments
Labels
l1-pending pending-signatures upgrade-pending

Comments

@makortel
Copy link
Contributor

makortel commented Apr 11, 2023 •
edited
Loading

ASAN reports in CMSSW_13_1_ASAN_X_2023-04-10-2300 workflow 23834.999 step 3

==26001==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x625005383900 at pc 0x2b759174aeba bp 0x7ffef4608e10 sp 0x7ffef4608e08
READ of size 2 at 0x625005383900 thread T0
    #0 0x2b759174aeb9 in ap_fixed_base<20, 15, true, (ap_q_mode)4, (ap_o_mode)0, 0>::RType<10, 1, false>::mult ap_fixed_base<20, 15, true, (ap_q_mode)4, (ap_o_mode)0, 0>::operator*<10, 1, false, (ap_q_mode)4, (ap_o_mode)0, 0>(ap_fixed_base<10, 1, false, (ap_q_mode)4, (ap_o_mode)0, 0> const&) const (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02780/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-10-2300/lib/el8_amd64_gcc11/pluginL1TrackTriggerPlugins.so+0x18beb9)
    #1 0x2b759175f2d7 in L1TrackerEtMissEmulatorProducer::produce(edm::Event&, edm::EventSetup const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02780/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-10-2300/lib/el8_amd64_gcc11/pluginL1TrackTriggerPlugins.so+0x1a02d7)
    #2 0x2b7502de3807 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02780/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-10-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x7a7807)
    #3 0x2b7502d795e0 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02780/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-10-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x73d5e0)
    #4 0x2b7502b2005c in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02780/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-10-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x4e405c)
    #5 0x2b7502b425ca in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02780/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-10-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x5065ca)
    #6 0x2b75037f66cb in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02780/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-10-2300/lib/el8_amd64_gcc11/libFWCoreConcurrency.so+0x96cb)
    #7 0x2b7504e579cc in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter>(tbb::detail::d1::task*, tbb::detail::r1::external_waiter&) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322
    #8 0x2b7504e579cc in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter>(tbb::detail::d1::task*, tbb::detail::r1::external_waiter&) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458
    #9 0x2b7504e579cc in tbb::detail::r1::task_dispatcher::execute_and_wait(tbb::detail::d1::task*, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/task_dispatcher.cpp:168
    #10 0x2b7502885d59 in edm::FinalWaitingTask::wait() (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02780/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-10-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x249d59)
    #11 0x2b75028e67b7 in edm::EventProcessor::processRuns() (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02780/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-10-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x2aa7b7)
    #12 0x2b75028c4c4d in edm::EventProcessor::runToCompletion() (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02780/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-10-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x288c4d)
    #13 0x4193a1 in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02780/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-10-2300/bin/el8_amd64_gcc11/cmsRun+0x4193a1)
    #14 0x2b7504e45846 in tbb::detail::r1::task_arena_impl::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/arena.cpp:694
    #15 0x42e3f8 in main::{lambda()#1}::operator()() const (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02780/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-10-2300/bin/el8_amd64_gcc11/cmsRun+0x42e3f8)
    #16 0x40be95 in main (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02780/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-10-2300/bin/el8_amd64_gcc11/cmsRun+0x40be95)
    #17 0x2b7505d4fd84 in __libc_start_main (/lib64/libc.so.6+0x3ad84)
    #18 0x40c57d in _start (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02780/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-10-2300/bin/el8_amd64_gcc11/cmsRun+0x40c57d)

0x625005383900 is located 0 bytes to the right of 8192-byte region [0x625005381900,0x625005383900)
allocated by thread T4 here:
    #0 0x2b7501d1af37 in operator new(unsigned long) ../../../../libsanitizer/asan/asan_new_delete.cpp:99
    #1 0x2b7591886d33 in ap_ufixed<10, 1, (ap_q_mode)4, (ap_o_mode)0, 0>& std::vector<ap_ufixed<10, 1, (ap_q_mode)4, (ap_o_mode)0, 0>, std::allocator<ap_ufixed<10, 1, (ap_q_mode)4, (ap_o_mode)0, 0> > >::emplace_back<ap_ufixed<10, 1, (ap_q_mode)4, (ap_o_mode)0, 0> >(ap_ufixed<10, 1, (ap_q_mode)4, (ap_o_mode)0, 0>&&) [clone .isra.0] (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02780/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-10-2300/lib/el8_amd64_gcc11/libL1TriggerL1TTrackMatch.so+0x48d33)
    #2 0x2b759186ee22 in l1tmetemu::generateCosLUT(unsigned int) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02780/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-10-2300/lib/el8_amd64_gcc11/libL1TriggerL1TTrackMatch.so+0x30e22)
    #3 0x2b759175506e in L1TrackerEtMissEmulatorProducer::L1TrackerEtMissEmulatorProducer(edm::ParameterSet const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02780/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-10-2300/lib/el8_amd64_gcc11/pluginL1TrackTriggerPlugins.so+0x19606e)
    #4 0x2b759178c8fd in edm::stream::ProducingModuleAdaptor<L1TrackerEtMissEmulatorProducer, edm::stream::EDProducerBase, edm::stream::EDProducerAdaptorBase>::setupStreamModules() (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02780/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-10-2300/lib/el8_amd64_gcc11/pluginL1TrackTriggerPlugins.so+0x1cd8fd)
    #5 0x2b7502e12688 in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doPreallocate(edm::PreallocationConfiguration const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02780/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-10-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x7d6688)
    #6 0x2b7502d62d16 in edm::Maker::makeModule(edm::MakeModuleParams const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02780/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-10-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x726d16)
    #7 0x2b7502a9cf09 in edm::Factory::makeModule(edm::MakeModuleParams const&, edm::ModuleTypeResolverMaker const*, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02780/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-10-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x460f09)
    #8 0x2b7502adfa58 in edm::ModuleRegistry::getModule(edm::MakeModuleParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02780/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-10-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x4a3a58)
    #9 0x2b7502d5572d in edm::WorkerRegistry::getWorker(edm::WorkerParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02780/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-10-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x71972d)
    #10 0x2b7502d5693d in edm::WorkerManager::getWorker(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02780/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-10-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x71a93d)
    #11 0x2b7502d57149 in edm::WorkerManager::addToUnscheduledWorkers(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02780/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-10-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x71b149)
    #12 0x2b7502cb5fbc in edm::StreamSchedule::StreamSchedule(std::shared_ptr<edm::TriggerResultInserter>, std::vector<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> > > >&, std::vector<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> > > >&, std::shared_ptr<edm::ModuleRegistry>, edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::PreallocationConfiguration const&, edm::ProductRegistry&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration const>, edm::StreamID, edm::ProcessContext const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02780/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-10-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x679fbc)
    #13 0x2b7502c3aa77 in edm::Schedule::Schedule(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::ProductRegistry&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration const>, edm::PreallocationConfiguration const&, edm::ProcessContext const*, edm::ModuleTypeResolverMaker const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02780/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-10-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x5fea77)
    #14 0x2b7502ca8b62 in edm::ScheduleItems::initModules(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::PreallocationConfiguration const&, edm::ProcessContext const*, edm::ModuleTypeResolverMaker const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02780/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-10-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x66cb62)
    #15 0x2b75029eab51 in tbb::detail::d1::function_task<edm::EventProcessor::init(std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) [clone .lto_priv.0] (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02780/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-10-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x3aeb51)
    #16 0x2b7504e50303 in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322
    #17 0x2b7504e50303 in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458
    #18 0x2b7504e50303 in tbb::detail::r1::arena::process(tbb::detail::r1::thread_data&) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/arena.cpp:137
    #19 0x2b7504e50303 in tbb::detail::r1::market::process(rml::job&) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/market.cpp:599
    #20 0x2b7504e524c5 in tbb::detail::r1::rml::private_worker::run() /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:271
    #21 0x2b7504e524c5 in tbb::detail::r1::rml::private_worker::thread_routine(void*) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:221

Thread T4 created by T0 here:
    #0 0x2b7501cc1716 in __interceptor_pthread_create ../../../../libsanitizer/asan/asan_interceptors.cpp:216
    #1 0x2b7504e51b9a in tbb::detail::r1::rml::internal::thread_monitor::launch(void* (*)(void*), void*, unsigned long) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/rml_thread_monitor.h:195
    #2 0x2b7504e51b9a in tbb::detail::r1::rml::private_worker::wake_or_launch() /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:305
    #3 0x2b7504e51b9a in tbb::detail::r1::rml::private_server::wake_some(int) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:412
    #4 0x2b750ae5107f  (<unknown module>)

SUMMARY: AddressSanitizer: heap-buffer-overflow (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02780/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-10-2300/lib/el8_amd64_gcc11/pluginL1TrackTriggerPlugins.so+0x18beb9) in ap_fixed_base<20, 15, true, (ap_q_mode)4, (ap_o_mode)0, 0>::RType<10, 1, false>::mult ap_fixed_base<20, 15, true, (ap_q_mode)4, (ap_o_mode)0, 0>::operator*<10, 1, false, (ap_q_mode)4, (ap_o_mode)0, 0>(ap_fixed_base<10, 1, false, (ap_q_mode)4, (ap_o_mode)0, 0> const&) const
Shadow bytes around the buggy address:
  0x0c4a80a686d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c4a80a686e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c4a80a686f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c4a80a68700: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c4a80a68710: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c4a80a68720:[fa]fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a80a68730: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a80a68740: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a80a68750: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a80a68760: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a80a68770: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el8_amd64_gcc11/CMSSW_13_1_ASAN_X_2023-04-10-2300/pyRelValMatrixLogs/run/23834.999_TTbar_14TeV+2026D95PU_PMXS1S2PR/step3_TTbar_14TeV+2026D95PU_PMXS1S2PR.log#/
@makortel
Copy link
Contributor Author

assign l1, upgrade

@cmsbuild
Copy link
Contributor

New categories assigned: upgrade,l1

@epalencia,@AdrianoDee,@srimanob,@aloeliger,@cecilecaillol you have been requested to review this Pull request/Issue and eventually sign? Thanks

@cmsbuild
Copy link
Contributor

A new Issue was created by @makortel Matti Kortelainen.

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

@makortel
Copy link
Contributor Author

In the earlier ASAN IB (CMSSW_13_1 2023-04-07-2300) the workflow succeeded. None of the PRs merged between two look like immediately clear culprit though.

@aloeliger
Copy link
Contributor

@makortel If I needed to recreate this error, what's the best way to go about it?

@aloeliger aloeliger mentioned this issue Apr 12, 2023
Open
@iarspider
Copy link
Contributor

@aloeliger on lxplus or cmsdev:

$ cmssw-el8
$ export SCRAM_ARCH=el8_amd64_gcc11
$ scram project CMSSW_13_1_ASAN_X_2023-04-10-2300
$ cd CMSSW_13_1_ASAN_X_2023-04-10-2300/src
$ cmsenv
$ git cms-addpkg L1Trigger/L1TTrackMatch
$ USER_CXXFLAGS='-g' scram b -j8
$ runTheMatrix.py -i all -l 23834.999 -t 4 --ibeos

@makortel
Copy link
Contributor Author

Interestingly the report did not get reproduced in CMSSW_13_1_ASAN_X_2023-04-12-2300

@aloeliger
Copy link
Contributor

Sorry, I finally had some time to look at this today. I can get @iarspider's instructions to run all the way through. Looking at the original output, should I be running this through valgrind or something similar to catch this issue?

@makortel
Copy link
Contributor Author

Looking at the original output, should I be running this through valgrind or something similar to catch this issue?

No, the Address Sanitizer is linked within the binaries in the ASAN build, and nothing special needs to be done to run them.

The error has not reproduced in IBs either. It could be that there is some rare non-reproducibility in play.

@aloeliger
Copy link
Contributor

I mean, as best I can understand the ASAN (address sanitizer? This is new to me. At the risk of sounding like I don't have a clue I'd be curious to hear or read more) stack trace output, the primary complaint is that L1TrackerEtMissEmulatorProducer::produce has a buffer overflow with the interaction/allocation made by generateCosLUT.

I looked that up and actually found two different implementations. I think the primary intended one in this scenario is this one:

cmssw/L1Trigger/L1TTrackMatch/src/L1TkEtMissEmuAlgo.cc

Lines 6 to 16 in e01b0cd

std::vector<cos_lut_fixed_t> generateCosLUT(unsigned int size) { // Fill cosine LUT with integer values
float phi = 0;
std::vector<cos_lut_fixed_t> cosLUT;
for (unsigned int LUT_idx = 0; LUT_idx < size; LUT_idx++) {
cosLUT.push_back((cos_lut_fixed_t)(cos(phi)));
phi += TTTrack_TrackWord::stepPhi0;
//std::cout << LUT_idx << "," << (cos_lut_fixed_t)(cos(phi)) << std::endl;
}
cosLUT.push_back((cos_lut_fixed_t)(0)); //Prevent overflow in last bin
return cosLUT;
}

Which returns a std::vector of cos_lut_fixed_t which are an ap_ufixed of the size mentioned in the debug: ap_ufixed<10, 1, (ap_q_mode)4, (ap_o_mode)0, 0>. Strictly, I don't seen anything here which would qualify as memory mismanagement. The casts are a little weird I guess, but having interacted with the ap types, I trust them to be able to handle that I guess?

If you look at where the cosLUT value is used, this is the primary use case:

cmssw/L1Trigger/L1TTrackMatch/plugins/L1TrackerEtMissEmulatorProducer.cc

Lines 184 to 226 in 5b979f2

// Split tracks in phi quadrants and access cosLUT_, backwards iteration
// through cosLUT_ gives sin Sum sector Et -ve when cos or sin phi are -ve
sector_totals[track->phiSector()] += 1;
if (globalPhi >= phiQuadrants_[0] && globalPhi < phiQuadrants_[1]) {
temppx = ((l1tmetemu::Et_t)ptEmulation * cosLUT_[globalPhi]);
temppy = ((l1tmetemu::Et_t)ptEmulation * cosLUT_[phiQuadrants_[1] - 1 - globalPhi]);
if (debug_ == 2) {
edm::LogVerbatim("L1TrackerEtMissEmulatorProducer")
<< "Sector: " << track->phiSector() << " Quadrant: " << 1 << "\n"
<< "Emu Phi: " << globalPhi << " Emu Cos(Phi): " << cosLUT_[globalPhi]
<< " Emu Sin(Phi): " << cosLUT_[phiQuadrants_[1] - 1 - globalPhi] << "\n";
}
} else if (globalPhi >= phiQuadrants_[1] && globalPhi < phiQuadrants_[2]) {
temppx = -((l1tmetemu::Et_t)ptEmulation * cosLUT_[phiQuadrants_[2] - 1 - globalPhi]);
temppy = ((l1tmetemu::Et_t)ptEmulation * cosLUT_[globalPhi - phiQuadrants_[1]]);
if (debug_ == 2) {
edm::LogVerbatim("L1TrackerEtMissEmulatorProducer")
<< "Sector: " << track->phiSector() << " Quadrant: " << 2 << "\n"
<< "Emu Phi: " << globalPhi << " Emu Cos(Phi): -" << cosLUT_[phiQuadrants_[2] - 1 - globalPhi]
<< " Emu Sin(Phi): " << cosLUT_[globalPhi - phiQuadrants_[1]] << "\n";
}
} else if (globalPhi >= phiQuadrants_[2] && globalPhi < phiQuadrants_[3]) {
temppx = -((l1tmetemu::Et_t)ptEmulation * cosLUT_[globalPhi - phiQuadrants_[2]]);
temppy = -((l1tmetemu::Et_t)ptEmulation * cosLUT_[phiQuadrants_[3] - 1 - globalPhi]);
if (debug_ == 2) {
edm::LogVerbatim("L1TrackerEtMissEmulatorProducer")
<< "Sector: " << track->phiSector() << " Quadrant: " << 3 << "\n"
<< "Emu Phi: " << globalPhi << " Emu Cos(Phi): -" << cosLUT_[globalPhi - phiQuadrants_[2]]
<< " Emu Sin(Phi): -" << cosLUT_[phiQuadrants_[3] - 1 - globalPhi] << "\n";
}
} else if (globalPhi >= phiQuadrants_[3] && globalPhi < phiQuadrants_[4]) {
temppx = ((l1tmetemu::Et_t)ptEmulation * cosLUT_[phiQuadrants_[4] - 1 - globalPhi]);
temppy = -((l1tmetemu::Et_t)ptEmulation * cosLUT_[globalPhi - phiQuadrants_[3]]);
if (debug_ == 2) {
edm::LogVerbatim("L1TrackerEtMissEmulatorProducer")
<< "Sector: " << track->phiSector() << " Quadrant: " << 4 << "\n"
<< " Emu Phi: " << globalPhi << " Emu Cos(Phi): " << cosLUT_[phiQuadrants_[4] - 1 - globalPhi]
<< " Emu Sin(Phi): -" << cosLUT_[globalPhi - phiQuadrants_[3]] << "\n";

But this doesn't seem like it would have any non-deterministic problem in it. I'm also not seeing an obvious ap_fixed_base<20, 15, true, (ap_q_mode)4, (ap_o_mode)0, 0> for there to be an issue with.

@makortel
Copy link
Contributor Author

Another occurrence in CMSSW_13_1_ASAN_X_2023-04-24-2300

==23506==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x625001efc100 at pc 0x2ab93d7b3eba bp 0x2ab919580990 sp 0x2ab919580988
READ of size 2 at 0x625001efc100 thread T4
    #0 0x2ab93d7b3eb9 in ap_fixed_base<20, 15, true, (ap_q_mode)4, (ap_o_mode)0, 0>::RType<10, 1, false>::mult ap_fixed_base<20, 15, true, (ap_q_mode)4, (ap_o_mode)0, 0>::operator*<10, 1, false, (ap_q_mode)4, (ap_o_mode)0, 0>(ap_fixed_base<10, 1, false, (ap_q_mode)4, (ap_o_mode)0, 0> const&) const (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02782/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-24-2300/lib/el8_amd64_gcc11/pluginL1TrackTriggerPlugins.so+0x18beb9)
    #1 0x2ab93d7c82d7 in L1TrackerEtMissEmulatorProducer::produce(edm::Event&, edm::EventSetup const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02782/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-24-2300/lib/el8_amd64_gcc11/pluginL1TrackTriggerPlugins.so+0x1a02d7)
    #2 0x2ab8af70d807 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02782/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-24-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x7a7807)
    #3 0x2ab8af6a35e0 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02782/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-24-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x73d5e0)
    #4 0x2ab8af44a05c in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02782/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-24-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x4e405c)
    #5 0x2ab8af46c5ca in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02782/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-24-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x5065ca)
    #6 0x2ab8b01206cb in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02782/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-24-2300/lib/el8_amd64_gcc11/libFWCoreConcurrency.so+0x96cb)
    #7 0x2ab8b177a303 in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322
    #8 0x2ab8b177a303 in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458
    #9 0x2ab8b177a303 in tbb::detail::r1::arena::process(tbb::detail::r1::thread_data&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/arena.cpp:137
    #10 0x2ab8b177a303 in tbb::detail::r1::market::process(rml::job&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/market.cpp:599
    #11 0x2ab8b177c4c5 in tbb::detail::r1::rml::private_worker::run() /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:271
    #12 0x2ab8b177c4c5 in tbb::detail::r1::rml::private_worker::thread_routine(void*) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:221
    #13 0x2ab8b24271ce in start_thread (/lib64/libpthread.so.0+0x81ce)
    #14 0x2ab8b2678e72 in clone (/lib64/libc.so.6+0x39e72)

0x625001efc100 is located 0 bytes to the right of 8192-byte region [0x625001efa100,0x625001efc100)
allocated by thread T2 here:
    #0 0x2ab8ae644f37 in operator new(unsigned long) ../../../../libsanitizer/asan/asan_new_delete.cpp:99
    #1 0x2ab93d8efd33 in ap_ufixed<10, 1, (ap_q_mode)4, (ap_o_mode)0, 0>& std::vector<ap_ufixed<10, 1, (ap_q_mode)4, (ap_o_mode)0, 0>, std::allocator<ap_ufixed<10, 1, (ap_q_mode)4, (ap_o_mode)0, 0> > >::emplace_back<ap_ufixed<10, 1, (ap_q_mode)4, (ap_o_mode)0, 0> >(ap_ufixed<10, 1, (ap_q_mode)4, (ap_o_mode)0, 0>&&) [clone .isra.0] (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02782/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-24-2300/lib/el8_amd64_gcc11/libL1TriggerL1TTrackMatch.so+0x48d33)
    #2 0x2ab93d8d7e22 in l1tmetemu::generateCosLUT(unsigned int) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02782/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-24-2300/lib/el8_amd64_gcc11/libL1TriggerL1TTrackMatch.so+0x30e22)
    #3 0x2ab93d7be06e in L1TrackerEtMissEmulatorProducer::L1TrackerEtMissEmulatorProducer(edm::ParameterSet const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02782/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-24-2300/lib/el8_amd64_gcc11/pluginL1TrackTriggerPlugins.so+0x19606e)
    #4 0x2ab93d7f58fd in edm::stream::ProducingModuleAdaptor<L1TrackerEtMissEmulatorProducer, edm::stream::EDProducerBase, edm::stream::EDProducerAdaptorBase>::setupStreamModules() (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02782/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-24-2300/lib/el8_amd64_gcc11/pluginL1TrackTriggerPlugins.so+0x1cd8fd)
    #5 0x2ab8af73c688 in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doPreallocate(edm::PreallocationConfiguration const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02782/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-24-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x7d6688)
    #6 0x2ab8af68cd16 in edm::Maker::makeModule(edm::MakeModuleParams const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02782/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-24-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x726d16)
    #7 0x2ab8af3c6f09 in edm::Factory::makeModule(edm::MakeModuleParams const&, edm::ModuleTypeResolverMaker const*, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02782/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-24-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x460f09)
    #8 0x2ab8af409a58 in edm::ModuleRegistry::getModule(edm::MakeModuleParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02782/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-24-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x4a3a58)
    #9 0x2ab8af67f72d in edm::WorkerRegistry::getWorker(edm::WorkerParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02782/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-24-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x71972d)
    #10 0x2ab8af68093d in edm::WorkerManager::getWorker(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02782/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-24-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x71a93d)
    #11 0x2ab8af681149 in edm::WorkerManager::addToUnscheduledWorkers(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02782/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-24-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x71b149)
    #12 0x2ab8af5dffbc in edm::StreamSchedule::StreamSchedule(std::shared_ptr<edm::TriggerResultInserter>, std::vector<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> > > >&, std::vector<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> > > >&, std::shared_ptr<edm::ModuleRegistry>, edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::PreallocationConfiguration const&, edm::ProductRegistry&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration const>, edm::StreamID, edm::ProcessContext const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02782/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-24-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x679fbc)
    #13 0x2ab8af564a77 in edm::Schedule::Schedule(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::ProductRegistry&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration const>, edm::PreallocationConfiguration const&, edm::ProcessContext const*, edm::ModuleTypeResolverMaker const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02782/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-24-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x5fea77)
    #14 0x2ab8af5d2b62 in edm::ScheduleItems::initModules(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::PreallocationConfiguration const&, edm::ProcessContext const*, edm::ModuleTypeResolverMaker const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02782/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-24-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x66cb62)
    #15 0x2ab8af314b51 in tbb::detail::d1::function_task<edm::EventProcessor::init(std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) [clone .lto_priv.0] (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02782/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-24-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x3aeb51)
    #16 0x2ab8b177a303 in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322
    #17 0x2ab8b177a303 in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458
    #18 0x2ab8b177a303 in tbb::detail::r1::arena::process(tbb::detail::r1::thread_data&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/arena.cpp:137
    #19 0x2ab8b177a303 in tbb::detail::r1::market::process(rml::job&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/market.cpp:599
    #20 0x2ab8b177c4c5 in tbb::detail::r1::rml::private_worker::run() /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:271
    #21 0x2ab8b177c4c5 in tbb::detail::r1::rml::private_worker::thread_routine(void*) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:221

Thread T4 created by T0 here:
    #0 0x2ab8ae5eb716 in __interceptor_pthread_create ../../../../libsanitizer/asan/asan_interceptors.cpp:216
    #1 0x2ab8b177bb9a in tbb::detail::r1::rml::internal::thread_monitor::launch(void* (*)(void*), void*, unsigned long) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/rml_thread_monitor.h:195
    #2 0x2ab8b177bb9a in tbb::detail::r1::rml::private_worker::wake_or_launch() /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:305
    #3 0x2ab8b177bb9a in tbb::detail::r1::rml::private_server::wake_some(int) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:412
    #4 0x2ab8b777b07f  (<unknown module>)

Thread T2 created by T0 here:
    #0 0x2ab8ae5eb716 in __interceptor_pthread_create ../../../../libsanitizer/asan/asan_interceptors.cpp:216
    #1 0x2ab8b177bb9a in tbb::detail::r1::rml::internal::thread_monitor::launch(void* (*)(void*), void*, unsigned long) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/rml_thread_monitor.h:195
    #2 0x2ab8b177bb9a in tbb::detail::r1::rml::private_worker::wake_or_launch() /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:305
    #3 0x2ab8b177bb9a in tbb::detail::r1::rml::private_server::wake_some(int) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:412
    #4 0x60c000426b7f  (<unknown module>)

SUMMARY: AddressSanitizer: heap-buffer-overflow (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02782/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ASAN_X_2023-04-24-2300/lib/el8_amd64_gcc11/pluginL1TrackTriggerPlugins.so+0x18beb9) in ap_fixed_base<20, 15, true, (ap_q_mode)4, (ap_o_mode)0, 0>::RType<10, 1, false>::mult ap_fixed_base<20, 15, true, (ap_q_mode)4, (ap_o_mode)0, 0>::operator*<10, 1, false, (ap_q_mode)4, (ap_o_mode)0, 0>(ap_fixed_base<10, 1, false, (ap_q_mode)4, (ap_o_mode)0, 0> const&) const
Shadow bytes around the buggy address:
  0x0c4a803d77d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c4a803d77e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c4a803d77f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c4a803d7800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c4a803d7810: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c4a803d7820:[fa]fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a803d7830: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a803d7840: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a803d7850: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a803d7860: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a803d7870: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el8_amd64_gcc11/CMSSW_13_1_ASAN_X_2023-04-24-2300/pyRelValMatrixLogs/run/23834.999_TTbar_14TeV+2026D95PU_PMXS1S2PR/step3_TTbar_14TeV+2026D95PU_PMXS1S2PR.log#/

@makortel
Copy link
Contributor Author

Another occurrence in CMSSW_13_2_ASAN_X_2023-05-17-2300

==25288==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6250075be100 at pc 0x2ab1691966aa bp 0x2ab13e09dd50 sp 0x2ab13e09dd48
READ of size 2 at 0x6250075be100 thread T3
    #0 0x2ab1691966a9 in ap_fixed_base<20, 15, true, (ap_q_mode)4, (ap_o_mode)0, 0>::RType<10, 1, false>::mult ap_fixed_base<20, 15, true, (ap_q_mode)4, (ap_o_mode)0, 0>::operator*<10, 1, false, (ap_q_mode)4, (ap_o_mode)0, 0>(ap_fixed_base<10, 1, false, (ap_q_mode)4, (ap_o_mode)0, 0> const&) const (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02785/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-17-2300/lib/el8_amd64_gcc11/pluginL1TrackTriggerPlugins.so+0x1a76a9)
    #1 0x2ab1691b08b7 in L1TrackerEtMissEmulatorProducer::produce(edm::Event&, edm::EventSetup const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02785/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-17-2300/lib/el8_amd64_gcc11/pluginL1TrackTriggerPlugins.so+0x1c18b7)
    #2 0x2ab0d4cf5547 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02785/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-17-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x7aa547)
    #3 0x2ab0d4c8a370 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02785/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-17-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x73f370)
    #4 0x2ab0d4a3139c in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02785/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-17-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x4e639c)
    #5 0x2ab0d4a52c6a in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02785/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-17-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x507c6a)
    #6 0x2ab0d3b406cb in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02785/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-17-2300/lib/el8_amd64_gcc11/libFWCoreConcurrency.so+0x96cb)
    #7 0x2ab0d6d07303 in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322
    #8 0x2ab0d6d07303 in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458
    #9 0x2ab0d6d07303 in tbb::detail::r1::arena::process(tbb::detail::r1::thread_data&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/arena.cpp:137
    #10 0x2ab0d6d07303 in tbb::detail::r1::market::process(rml::job&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/market.cpp:599
    #11 0x2ab0d6d094c5 in tbb::detail::r1::rml::private_worker::run() /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:271
    #12 0x2ab0d6d094c5 in tbb::detail::r1::rml::private_worker::thread_routine(void*) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:221
    #13 0x2ab0d79b21ce in start_thread (/lib64/libpthread.so.0+0x81ce)
    #14 0x2ab0d7c03e72 in clone (/lib64/libc.so.6+0x39e72)

0x6250075be100 is located 0 bytes to the right of 8192-byte region [0x6250075bc100,0x6250075be100)
allocated by thread T2 here:
    #0 0x2ab0d3c29f37 in operator new(unsigned long) ../../../../libsanitizer/asan/asan_new_delete.cpp:99
    #1 0x2ab1692d7d33 in ap_ufixed<10, 1, (ap_q_mode)4, (ap_o_mode)0, 0>& std::vector<ap_ufixed<10, 1, (ap_q_mode)4, (ap_o_mode)0, 0>, std::allocator<ap_ufixed<10, 1, (ap_q_mode)4, (ap_o_mode)0, 0> > >::emplace_back<ap_ufixed<10, 1, (ap_q_mode)4, (ap_o_mode)0, 0> >(ap_ufixed<10, 1, (ap_q_mode)4, (ap_o_mode)0, 0>&&) [clone .isra.0] (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02785/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-17-2300/lib/el8_amd64_gcc11/libL1TriggerL1TTrackMatch.so+0x48d33)
    #2 0x2ab1692bfe22 in l1tmetemu::generateCosLUT(unsigned int) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02785/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-17-2300/lib/el8_amd64_gcc11/libL1TriggerL1TTrackMatch.so+0x30e22)
    #3 0x2ab16917f728 in L1TrackerEtMissEmulatorProducer::L1TrackerEtMissEmulatorProducer(edm::ParameterSet const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02785/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-17-2300/lib/el8_amd64_gcc11/pluginL1TrackTriggerPlugins.so+0x190728)
    #4 0x2ab1691e606d in edm::stream::ProducingModuleAdaptor<L1TrackerEtMissEmulatorProducer, edm::stream::EDProducerBase, edm::stream::EDProducerAdaptorBase>::setupStreamModules() (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02785/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-17-2300/lib/el8_amd64_gcc11/pluginL1TrackTriggerPlugins.so+0x1f706d)
    #5 0x2ab0d4ce59e8 in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doPreallocate(edm::PreallocationConfiguration const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02785/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-17-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x79a9e8)
    #6 0x2ab0d4c73aa6 in edm::Maker::makeModule(edm::MakeModuleParams const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02785/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-17-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x728aa6)
    #7 0x2ab0d49abfc9 in edm::Factory::makeModule(edm::MakeModuleParams const&, edm::ModuleTypeResolverMaker const*, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02785/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-17-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x460fc9)
    #8 0x2ab0d49eeb18 in edm::ModuleRegistry::getModule(edm::MakeModuleParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02785/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-17-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x4a3b18)
    #9 0x2ab0d4c664bd in edm::WorkerRegistry::getWorker(edm::WorkerParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02785/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-17-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x71b4bd)
    #10 0x2ab0d4c676cd in edm::WorkerManager::getWorker(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02785/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-17-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x71c6cd)
    #11 0x2ab0d4c67ed9 in edm::WorkerManager::addToUnscheduledWorkers(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02785/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-17-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x71ced9)
    #12 0x2ab0d4bc6ecc in edm::StreamSchedule::StreamSchedule(std::shared_ptr<edm::TriggerResultInserter>, std::vector<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> > > >&, std::vector<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> > > >&, std::shared_ptr<edm::ModuleRegistry>, edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::PreallocationConfiguration const&, edm::ProductRegistry&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration const>, edm::StreamID, edm::ProcessContext const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02785/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-17-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x67becc)
    #13 0x2ab0d4b4b987 in edm::Schedule::Schedule(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::ProductRegistry&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration const>, edm::PreallocationConfiguration const&, edm::ProcessContext const*, edm::ModuleTypeResolverMaker const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02785/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-17-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x600987)
    #14 0x2ab0d4bb9a72 in edm::ScheduleItems::initModules(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::PreallocationConfiguration const&, edm::ProcessContext const*, edm::ModuleTypeResolverMaker const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02785/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-17-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x66ea72)
    #15 0x2ab0d48f9cb1 in tbb::detail::d1::function_task<edm::EventProcessor::init(std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) [clone .lto_priv.0] (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02785/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-17-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x3aecb1)
    #16 0x2ab0d6d07303 in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322
    #17 0x2ab0d6d07303 in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458
    #18 0x2ab0d6d07303 in tbb::detail::r1::arena::process(tbb::detail::r1::thread_data&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/arena.cpp:137
    #19 0x2ab0d6d07303 in tbb::detail::r1::market::process(rml::job&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/market.cpp:599
    #20 0x2ab0d6d094c5 in tbb::detail::r1::rml::private_worker::run() /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:271
    #21 0x2ab0d6d094c5 in tbb::detail::r1::rml::private_worker::thread_routine(void*) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:221

Thread T3 created by T2 here:
    #0 0x2ab0d3bd0716 in __interceptor_pthread_create ../../../../libsanitizer/asan/asan_interceptors.cpp:216
    #1 0x2ab0d6d08b9a in tbb::detail::r1::rml::internal::thread_monitor::launch(void* (*)(void*), void*, unsigned long) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/rml_thread_monitor.h:195
    #2 0x2ab0d6d08b9a in tbb::detail::r1::rml::private_worker::wake_or_launch() /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:305
    #3 0x2ab0d6d08b9a in tbb::detail::r1::rml::private_server::wake_some(int) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:412
    #4 0x7fff  (<unknown module>)

Thread T2 created by T0 here:
    #0 0x2ab0d3bd0716 in __interceptor_pthread_create ../../../../libsanitizer/asan/asan_interceptors.cpp:216
    #1 0x2ab0d6d08b9a in tbb::detail::r1::rml::internal::thread_monitor::launch(void* (*)(void*), void*, unsigned long) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/rml_thread_monitor.h:195
    #2 0x2ab0d6d08b9a in tbb::detail::r1::rml::private_worker::wake_or_launch() /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:305
    #3 0x2ab0d6d08b9a in tbb::detail::r1::rml::private_server::wake_some(int) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:412
    #4 0x60c0004284ff  (<unknown module>)

SUMMARY: AddressSanitizer: heap-buffer-overflow (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02785/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-17-2300/lib/el8_amd64_gcc11/pluginL1TrackTriggerPlugins.so+0x1a76a9) in ap_fixed_base<20, 15, true, (ap_q_mode)4, (ap_o_mode)0, 0>::RType<10, 1, false>::mult ap_fixed_base<20, 15, true, (ap_q_mode)4, (ap_o_mode)0, 0>::operator*<10, 1, false, (ap_q_mode)4, (ap_o_mode)0, 0>(ap_fixed_base<10, 1, false, (ap_q_mode)4, (ap_o_mode)0, 0> const&) const
Shadow bytes around the buggy address:
  0x0c4a80eafbd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c4a80eafbe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c4a80eafbf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c4a80eafc00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c4a80eafc10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c4a80eafc20:[fa]fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a80eafc30: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a80eafc40: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a80eafc50: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a80eafc60: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a80eafc70: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el8_amd64_gcc11/CMSSW_13_2_ASAN_X_2023-05-17-2300/pyRelValMatrixLogs/run/23834.999_TTbar_14TeV+2026D95PU_PMXS1S2PR/step3_TTbar_14TeV+2026D95PU_PMXS1S2PR.log#/

@aandvalenzuela
Copy link
Contributor

aandvalenzuela commented May 23, 2023 •
edited
Loading

Another occurrence in CMSSW_13_2_X_2023-05-22-2300:

==9196==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x625008ce5900 at pc 0x2aae5aa506aa bp 0x2aae2ef82d50 sp 0x2aae2ef82d48
READ of size 2 at 0x625008ce5900 thread T2
%MSG-w MemoryCheck:  PoolOutputModule:FEVTDEBUGHLToutput  23-May-2023 07:00:19 CEST Run: 1 Event: 5501
MemoryCheck: module PoolOutputModule:FEVTDEBUGHLToutput VSIZE 2.09777e+07 11.5977 RSS 11895.2 136.016
%MSG
%MSG-w MemoryCheck:  EndPathStatusInserter:FEVTDEBUGHLToutput_step  23-May-2023 07:00:19 CEST Run: 1 Event: 5501
MemoryCheck: module EndPathStatusInserter:FEVTDEBUGHLToutput_step VSIZE 2.09777e+07 0 RSS 11896.1 0.929688
%MSG
%MSG-w MemoryCheck:  HLTrigReport:hltTrigReport 23-May-2023 07:00:19 CEST  Run: 1 Event: 5501
MemoryCheck: module HLTrigReport:hltTrigReport VSIZE 2.09777e+07 0 RSS 11896.4 0.257812
%MSG
%MSG-w MemoryCheck:  EndPathStatusInserter:HLTAnalyzerEndpath  23-May-2023 07:00:19 CEST Run: 1 Event: 5501
MemoryCheck: module EndPathStatusInserter:HLTAnalyzerEndpath VSIZE 2.09777e+07 0.0625 RSS 11896.5 0.15625
%MSG
Begin processing the 7th record. Run 1, Event 5504, LumiSection 56 on stream 2 at 23-May-2023 07:00:25.060 CEST
[2023-05-23 07:00:26.607441 +0200][Error  ][PostMaster        ] [eoscms.cern.ch:1094] Forcing error on disconnect: [ERROR] Operation interrupted.
%MSG-w MemoryCheck:  MixingModule:mix 23-May-2023 07:00:41 CEST  Run: 1 Event: 5504
MemoryCheck: module MixingModule:mix VSIZE 2.09777e+07 -55.5234 RSS 12010.7 114.188
%MSG
%MSG-w MemoryCheck:  SiPixelClusterProducer:siPixelClusters  23-May-2023 07:00:41 CEST Run: 1 Event: 5504
MemoryCheck: module SiPixelClusterProducer:siPixelClusters VSIZE 2.09777e+07 -55.0508 RSS 12012.2 1.51172
%MSG
%MSG-w MemoryCheck:  SiPixelRecHitConverter:siPixelRecHits  23-May-2023 07:00:41 CEST Run: 1 Event: 5504
MemoryCheck: module SiPixelRecHitConverter:siPixelRecHits VSIZE 2.09777e+07 -54.7617 RSS 12012.8 0.585938
%MSG
%MSG-w MemoryCheck:  HitPairEDProducer:pixelTracksHitDoublets  23-May-2023 07:00:41 CEST Run: 1 Event: 5504
MemoryCheck: module HitPairEDProducer:pixelTracksHitDoublets VSIZE 2.09777e+07 -53.2617 RSS 12013 0.246094
%MSG
%MSG-w MemoryCheck:  SiPixelClusterShapeCacheProducer:siPixelClusterShapeCache  23-May-2023 07:00:41 CEST Run: 1 Event: 5504
MemoryCheck: module SiPixelClusterShapeCacheProducer:siPixelClusterShapeCache VSIZE 2.09777e+07 -53.2812 RSS 12013.2 0.175781
%MSG
%MSG-w MemoryCheck:  PixelTrackProducer:pixelTracks 23-May-2023 07:00:41 CEST  Run: 1 Event: 5504
MemoryCheck: module PixelTrackProducer:pixelTracks VSIZE 2.09777e+07 -53.0938 RSS 12013.5 0.300781
%MSG
%MSG-w MemoryCheck:  PixelVertexProducer:pixelVertices  23-May-2023 07:00:41 CEST Run: 1 Event: 5504
MemoryCheck: module PixelVertexProducer:pixelVertices VSIZE 2.09777e+07 -53.0938 RSS 12013.8 0.25
%MSG
%MSG-w MemoryCheck:  SeedGeneratorFromProtoTracksEDProducer:initialStepSeeds  23-May-2023 07:00:41 CEST Run: 1 Event: 5504
MemoryCheck: module SeedGeneratorFromProtoTracksEDProducer:initialStepSeeds VSIZE 2.09777e+07 -53.0898 RSS 12014.1 0.375
%MSG
    #0 0x2aae5aa506a9 in ap_fixed_base<20, 15, true, (ap_q_mode)4, (ap_o_mode)0, 0>::RType<10, 1, false>::mult ap_fixed_base<20, 15, true, (ap_q_mode)4, (ap_o_mode)0, 0>::operator*<10, 1, false, (ap_q_mode)4, (ap_o_mode)0, 0>(ap_fixed_base<10, 1, false, (ap_q_mode)4, (ap_o_mode)0, 0> const&) const (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-22-2300/lib/el8_amd64_gcc11/pluginL1TrackTriggerPlugins.so+0x1a76a9)
    #1 0x2aae5aa6a8b7 in L1TrackerEtMissEmulatorProducer::produce(edm::Event&, edm::EventSetup const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-22-2300/lib/el8_amd64_gcc11/pluginL1TrackTriggerPlugins.so+0x1c18b7)
    #2 0x2aadc662b547 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-22-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x7aa547)
    #3 0x2aadc65c0370 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-22-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x73f370)
    #4 0x2aadc636739c in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-22-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x4e639c)
    #5 0x2aadc6388c6a in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-22-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x507c6a)
    #6 0x2aadc54766cb in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-22-2300/lib/el8_amd64_gcc11/libFWCoreConcurrency.so+0x96cb)
    #7 0x2aadc863d303 in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322
    #8 0x2aadc863d303 in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458
    #9 0x2aadc863d303 in tbb::detail::r1::arena::process(tbb::detail::r1::thread_data&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/arena.cpp:137
    #10 0x2aadc863d303 in tbb::detail::r1::market::process(rml::job&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/market.cpp:599
    #11 0x2aadc863f4c5 in tbb::detail::r1::rml::private_worker::run() /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:271
    #12 0x2aadc863f4c5 in tbb::detail::r1::rml::private_worker::thread_routine(void*) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:221
    #13 0x2aadc92e81ce in start_thread (/lib64/libpthread.so.0+0x81ce)
    #14 0x2aadc9539e72 in clone (/lib64/libc.so.6+0x39e72)

0x625008ce5900 is located 0 bytes to the right of 8192-byte region [0x625008ce3900,0x625008ce5900)
allocated by thread T2 here:
    #0 0x2aadc555ff37 in operator new(unsigned long) ../../../../libsanitizer/asan/asan_new_delete.cpp:99
    #1 0x2aae52670d33 in ap_ufixed<10, 1, (ap_q_mode)4, (ap_o_mode)0, 0>& std::vector<ap_ufixed<10, 1, (ap_q_mode)4, (ap_o_mode)0, 0>, std::allocator<ap_ufixed<10, 1, (ap_q_mode)4, (ap_o_mode)0, 0> > >::emplace_back<ap_ufixed<10, 1, (ap_q_mode)4, (ap_o_mode)0, 0> >(ap_ufixed<10, 1, (ap_q_mode)4, (ap_o_mode)0, 0>&&) [clone .isra.0] (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-22-2300/lib/el8_amd64_gcc11/libL1TriggerL1TTrackMatch.so+0x48d33)
    #2 0x2aae52658e22 in l1tmetemu::generateCosLUT(unsigned int) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-22-2300/lib/el8_amd64_gcc11/libL1TriggerL1TTrackMatch.so+0x30e22)
    #3 0x2aae5aa39728 in L1TrackerEtMissEmulatorProducer::L1TrackerEtMissEmulatorProducer(edm::ParameterSet const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-22-2300/lib/el8_amd64_gcc11/pluginL1TrackTriggerPlugins.so+0x190728)
    #4 0x2aae5aaa006d in edm::stream::ProducingModuleAdaptor<L1TrackerEtMissEmulatorProducer, edm::stream::EDProducerBase, edm::stream::EDProducerAdaptorBase>::setupStreamModules() (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-22-2300/lib/el8_amd64_gcc11/pluginL1TrackTriggerPlugins.so+0x1f706d)
    #5 0x2aadc661b9e8 in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doPreallocate(edm::PreallocationConfiguration const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-22-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x79a9e8)
    #6 0x2aadc65a9aa6 in edm::Maker::makeModule(edm::MakeModuleParams const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-22-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x728aa6)
    #7 0x2aadc62e1fc9 in edm::Factory::makeModule(edm::MakeModuleParams const&, edm::ModuleTypeResolverMaker const*, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-22-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x460fc9)
    #8 0x2aadc6324b18 in edm::ModuleRegistry::getModule(edm::MakeModuleParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-22-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x4a3b18)
    #9 0x2aadc659c4bd in edm::WorkerRegistry::getWorker(edm::WorkerParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-22-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x71b4bd)
    #10 0x2aadc659d6cd in edm::WorkerManager::getWorker(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-22-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x71c6cd)
    #11 0x2aadc659ded9 in edm::WorkerManager::addToUnscheduledWorkers(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-22-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x71ced9)
    #12 0x2aadc64fcecc in edm::StreamSchedule::StreamSchedule(std::shared_ptr<edm::TriggerResultInserter>, std::vector<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> > > >&, std::vector<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> > > >&, std::shared_ptr<edm::ModuleRegistry>, edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::PreallocationConfiguration const&, edm::ProductRegistry&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration const>, edm::StreamID, edm::ProcessContext const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-22-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x67becc)
    #13 0x2aadc6481987 in edm::Schedule::Schedule(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::ProductRegistry&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration const>, edm::PreallocationConfiguration const&, edm::ProcessContext const*, edm::ModuleTypeResolverMaker const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-22-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x600987)
    #14 0x2aadc64efa72 in edm::ScheduleItems::initModules(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::PreallocationConfiguration const&, edm::ProcessContext const*, edm::ModuleTypeResolverMaker const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-22-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x66ea72)
    #15 0x2aadc622fcb1 in tbb::detail::d1::function_task<edm::EventProcessor::init(std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) [clone .lto_priv.0] (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-22-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x3aecb1)
    #16 0x2aadc863d303 in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322
    #17 0x2aadc863d303 in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458
    #18 0x2aadc863d303 in tbb::detail::r1::arena::process(tbb::detail::r1::thread_data&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/arena.cpp:137
    #19 0x2aadc863d303 in tbb::detail::r1::market::process(rml::job&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/market.cpp:599
    #20 0x2aadc863f4c5 in tbb::detail::r1::rml::private_worker::run() /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:271
    #21 0x2aadc863f4c5 in tbb::detail::r1::rml::private_worker::thread_routine(void*) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:221

Thread T2 created by T0 here:
    #0 0x2aadc5506716 in __interceptor_pthread_create ../../../../libsanitizer/asan/asan_interceptors.cpp:216
    #1 0x2aadc863eb9a in tbb::detail::r1::rml::internal::thread_monitor::launch(void* (*)(void*), void*, unsigned long) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/rml_thread_monitor.h:195
    #2 0x2aadc863eb9a in tbb::detail::r1::rml::private_worker::wake_or_launch() /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:305
    #3 0x2aadc863eb9a in tbb::detail::r1::rml::private_server::wake_some(int) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:412
    #4 0x60c0004324bf  (<unknown module>)

SUMMARY: AddressSanitizer: heap-buffer-overflow (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-22-2300/lib/el8_amd64_gcc11/pluginL1TrackTriggerPlugins.so+0x1a76a9) in ap_fixed_base<20, 15, true, (ap_q_mode)4, (ap_o_mode)0, 0>::RType<10, 1, false>::mult ap_fixed_base<20, 15, true, (ap_q_mode)4, (ap_o_mode)0, 0>::operator*<10, 1, false, (ap_q_mode)4, (ap_o_mode)0, 0>(ap_fixed_base<10, 1, false, (ap_q_mode)4, (ap_o_mode)0, 0> const&) const
Shadow bytes around the buggy address:
  0x0c4a81194ad0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c4a81194ae0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c4a81194af0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c4a81194b00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c4a81194b10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c4a81194b20:[fa]fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a81194b30: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a81194b40: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a81194b50: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a81194b60: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a81194b70: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==9196==ABORTING

@iarspider
Copy link
Contributor

Another occurrence in CMSSW_13_2_ASAN_X_2023-05-24-2300:

=================================================================
==15346==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x625004c5d900 at pc 0x2abb2dec06aa bp 0x2abb023a7d50 sp 0x2abb023a7d48
READ of size 2 at 0x625004c5d900 thread T2
    #0 0x2abb2dec06a9 in ap_fixed_base<20, 15, true, (ap_q_mode)4, (ap_o_mode)0, 0>::RType<10, 1, false>::mult ap_fixed_base<20, 15, true, (ap_q_mode)4, (ap_o_mode)0, 0>::operator*<10, 1, false, (ap_q_mode)4, (ap_o_mode)0, 0>(ap_fixed_base<10, 1, false, (ap_q_mode)4, (ap_o_mode)0, 0> const&) const (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-24-2300/lib/el8_amd64_gcc11/pluginL1TrackTriggerPlugins.so+0x1a76a9)
    #1 0x2abb2deda8b7 in L1TrackerEtMissEmulatorProducer::produce(edm::Event&, edm::EventSetup const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-24-2300/lib/el8_amd64_gcc11/pluginL1TrackTriggerPlugins.so+0x1c18b7)
    #2 0x2aba99a9f547 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-24-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x7aa547)
    #3 0x2aba99a34370 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-24-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x73f370)
    #4 0x2aba997db39c in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-24-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x4e639c)
    #5 0x2aba997fcc6a in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-24-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x507c6a)
    #6 0x2aba988ea6cb in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-24-2300/lib/el8_amd64_gcc11/libFWCoreConcurrency.so+0x96cb)
    #7 0x2aba9bab2303 in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322
    #8 0x2aba9bab2303 in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458
    #9 0x2aba9bab2303 in tbb::detail::r1::arena::process(tbb::detail::r1::thread_data&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/arena.cpp:137
    #10 0x2aba9bab2303 in tbb::detail::r1::market::process(rml::job&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/market.cpp:599
    #11 0x2aba9bab44c5 in tbb::detail::r1::rml::private_worker::run() /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:271
    #12 0x2aba9bab44c5 in tbb::detail::r1::rml::private_worker::thread_routine(void*) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:221
    #13 0x2aba9c75d1ce in start_thread (/lib64/libpthread.so.0+0x81ce)
    #14 0x2aba9c9aee72 in clone (/lib64/libc.so.6+0x39e72)

0x625004c5d900 is located 0 bytes to the right of 8192-byte region [0x625004c5b900,0x625004c5d900)
allocated by thread T3 here:
    #0 0x2aba989d3f37 in operator new(unsigned long) ../../../../libsanitizer/asan/asan_new_delete.cpp:99
    #1 0x2abb2e001d33 in ap_ufixed<10, 1, (ap_q_mode)4, (ap_o_mode)0, 0>& std::vector<ap_ufixed<10, 1, (ap_q_mode)4, (ap_o_mode)0, 0>, std::allocator<ap_ufixed<10, 1, (ap_q_mode)4, (ap_o_mode)0, 0> > >::emplace_back<ap_ufixed<10, 1, (ap_q_mode)4, (ap_o_mode)0, 0> >(ap_ufixed<10, 1, (ap_q_mode)4, (ap_o_mode)0, 0>&&) [clone .isra.0] (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-24-2300/lib/el8_amd64_gcc11/libL1TriggerL1TTrackMatch.so+0x48d33)
    #2 0x2abb2dfe9e22 in l1tmetemu::generateCosLUT(unsigned int) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-24-2300/lib/el8_amd64_gcc11/libL1TriggerL1TTrackMatch.so+0x30e22)
    #3 0x2abb2dea9728 in L1TrackerEtMissEmulatorProducer::L1TrackerEtMissEmulatorProducer(edm::ParameterSet const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-24-2300/lib/el8_amd64_gcc11/pluginL1TrackTriggerPlugins.so+0x190728)
    #4 0x2abb2df1006d in edm::stream::ProducingModuleAdaptor<L1TrackerEtMissEmulatorProducer, edm::stream::EDProducerBase, edm::stream::EDProducerAdaptorBase>::setupStreamModules() (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-24-2300/lib/el8_amd64_gcc11/pluginL1TrackTriggerPlugins.so+0x1f706d)
    #5 0x2aba99a8f9e8 in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doPreallocate(edm::PreallocationConfiguration const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-24-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x79a9e8)
    #6 0x2aba99a1daa6 in edm::Maker::makeModule(edm::MakeModuleParams const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-24-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x728aa6)
    #7 0x2aba99755fc9 in edm::Factory::makeModule(edm::MakeModuleParams const&, edm::ModuleTypeResolverMaker const*, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-24-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x460fc9)
    #8 0x2aba99798b18 in edm::ModuleRegistry::getModule(edm::MakeModuleParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-24-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x4a3b18)
    #9 0x2aba99a104bd in edm::WorkerRegistry::getWorker(edm::WorkerParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-24-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x71b4bd)
    #10 0x2aba99a116cd in edm::WorkerManager::getWorker(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-24-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x71c6cd)
    #11 0x2aba99a11ed9 in edm::WorkerManager::addToUnscheduledWorkers(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-24-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x71ced9)
    #12 0x2aba99970ecc in edm::StreamSchedule::StreamSchedule(std::shared_ptr<edm::TriggerResultInserter>, std::vector<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> > > >&, std::vector<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> > > >&, std::shared_ptr<edm::ModuleRegistry>, edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::PreallocationConfiguration const&, edm::ProductRegistry&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration const>, edm::StreamID, edm::ProcessContext const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-24-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x67becc)
    #13 0x2aba998f5987 in edm::Schedule::Schedule(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::ProductRegistry&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration const>, edm::PreallocationConfiguration const&, edm::ProcessContext const*, edm::ModuleTypeResolverMaker const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-24-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x600987)
    #14 0x2aba99963a72 in edm::ScheduleItems::initModules(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::PreallocationConfiguration const&, edm::ProcessContext const*, edm::ModuleTypeResolverMaker const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-24-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x66ea72)
    #15 0x2aba996a3cb1 in tbb::detail::d1::function_task<edm::EventProcessor::init(std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) [clone .lto_priv.0] (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-24-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x3aecb1)
    #16 0x2aba9bab2303 in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322
    #17 0x2aba9bab2303 in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458
    #18 0x2aba9bab2303 in tbb::detail::r1::arena::process(tbb::detail::r1::thread_data&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/arena.cpp:137
    #19 0x2aba9bab2303 in tbb::detail::r1::market::process(rml::job&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/market.cpp:599
    #20 0x2aba9bab44c5 in tbb::detail::r1::rml::private_worker::run() /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:271
    #21 0x2aba9bab44c5 in tbb::detail::r1::rml::private_worker::thread_routine(void*) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:221

Thread T2 created by T0 here:
    #0 0x2aba9897a716 in __interceptor_pthread_create ../../../../libsanitizer/asan/asan_interceptors.cpp:216
    #1 0x2aba9bab3b9a in tbb::detail::r1::rml::internal::thread_monitor::launch(void* (*)(void*), void*, unsigned long) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/rml_thread_monitor.h:195
    #2 0x2aba9bab3b9a in tbb::detail::r1::rml::private_worker::wake_or_launch() /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:305
    #3 0x2aba9bab3b9a in tbb::detail::r1::rml::private_server::wake_some(int) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:412
    #4 0x60c00042987f  (<unknown module>)

Thread T3 created by T0 here:
    #0 0x2aba9897a716 in __interceptor_pthread_create ../../../../libsanitizer/asan/asan_interceptors.cpp:216
    #1 0x2aba9bab3b9a in tbb::detail::r1::rml::internal::thread_monitor::launch(void* (*)(void*), void*, unsigned long) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/rml_thread_monitor.h:195
    #2 0x2aba9bab3b9a in tbb::detail::r1::rml::private_worker::wake_or_launch() /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:305
    #3 0x2aba9bab3b9a in tbb::detail::r1::rml::private_server::wake_some(int) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:412
    #4 0x2abaa1d9c07f  (<unknown module>)

SUMMARY: AddressSanitizer: heap-buffer-overflow (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02786/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-05-24-2300/lib/el8_amd64_gcc11/pluginL1TrackTriggerPlugins.so+0x1a76a9) in ap_fixed_base<20, 15, true, (ap_q_mode)4, (ap_o_mode)0, 0>::RType<10, 1, false>::mult ap_fixed_base<20, 15, true, (ap_q_mode)4, (ap_o_mode)0, 0>::operator*<10, 1, false, (ap_q_mode)4, (ap_o_mode)0, 0>(ap_fixed_base<10, 1, false, (ap_q_mode)4, (ap_o_mode)0, 0> const&) const
Shadow bytes around the buggy address:
  0x0c4a80983ad0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c4a80983ae0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c4a80983af0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c4a80983b00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c4a80983b10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c4a80983b20:[fa]fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a80983b30: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a80983b40: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a80983b50: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a80983b60: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a80983b70: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==15346==ABORTING

@aloeliger
Copy link
Contributor

aloeliger commented May 25, 2023 •
edited
Loading

@aperloff @Chriisbrown You are both listed as authors on L1TrackerEtMissEmulatorProducer and l1tmetemu::generateCosLUT which seems to be creating these ASAN errors. Could I ask you to please take a look?

@makortel
Copy link
Contributor Author

Another occurrence in CMSSW_13_2_ASAN_X_2023-06-09-2300 workflow 23834.999 step 3

=================================================================
==231397==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62500ad86100 at pc 0x146ba581c6aa bp 0x146bd1e4ed50 sp 0x146bd1e4ed48
    #0 0x146ba581c6a9 in ap_fixed_base<20, 15, true, (ap_q_mode)4, (ap_o_mode)0, 0>::RType<10, 1, false>::mult ap_fixed_base<20, 15, true, (ap_q_mode)4, (ap_o_mode)0, 0>::operator*<10, 1, false, (ap_q_mode)4, (ap_o_mode)0, 0>(ap_fixed_base<10, 1, false, (ap_q_mode)4, (ap_o_mode)0, 0> const&) const (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02788/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-06-09-2300/lib/el8_amd64_gcc11/pluginL1TrackTriggerPlugins.so+0x1a76a9)
    #1 0x146ba58368b7 in L1TrackerEtMissEmulatorProducer::produce(edm::Event&, edm::EventSetup const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02788/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-06-09-2300/lib/el8_amd64_gcc11/pluginL1TrackTriggerPlugins.so+0x1c18b7)
    #2 0x146c3a20c547 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02788/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-06-09-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x7aa547)
    #3 0x146c3a1a1370 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02788/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-06-09-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x73f370)
    #4 0x146c39f4839c in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02788/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-06-09-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x4e639c)
    #5 0x146c39f69c6a in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02788/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-06-09-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x507c6a)
    #6 0x146c3aee16cb in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02788/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-06-09-2300/lib/el8_amd64_gcc11/libFWCoreConcurrency.so+0x96cb)
    #7 0x146c37ced303 in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322
    #8 0x146c37ced303 in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458
    #9 0x146c37ced303 in tbb::detail::r1::arena::process(tbb::detail::r1::thread_data&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/arena.cpp:137
    #10 0x146c37ced303 in tbb::detail::r1::market::process(rml::job&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/market.cpp:599
    #11 0x146c37cef4c5 in tbb::detail::r1::rml::private_worker::run() /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:271
    #12 0x146c37cef4c5 in tbb::detail::r1::rml::private_worker::thread_routine(void*) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:221
    #13 0x146c36e421c9 in start_thread (/lib64/libpthread.so.0+0x81c9)
    #14 0x146c36aaee72 in __clone (/lib64/libc.so.6+0x39e72)

0x62500ad86100 is located 0 bytes to the right of 8192-byte region [0x62500ad84100,0x62500ad86100)
allocated by thread T4 here:
    #0 0x146c3a569f37 in operator new(unsigned long) ../../../../libsanitizer/asan/asan_new_delete.cpp:99
    #1 0x146bab918d33 in ap_ufixed<10, 1, (ap_q_mode)4, (ap_o_mode)0, 0>& std::vector<ap_ufixed<10, 1, (ap_q_mode)4, (ap_o_mode)0, 0>, std::allocator<ap_ufixed<10, 1, (ap_q_mode)4, (ap_o_mode)0, 0> > >::emplace_back<ap_ufixed<10, 1, (ap_q_mode)4, (ap_o_mode)0, 0> >(ap_ufixed<10, 1, (ap_q_mode)4, (ap_o_mode)0, 0>&&) [clone .isra.0] (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02788/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-06-09-2300/lib/el8_amd64_gcc11/libL1TriggerL1TTrackMatch.so+0x48d33)
    #2 0x146bab900e22 in l1tmetemu::generateCosLUT(unsigned int) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02788/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-06-09-2300/lib/el8_amd64_gcc11/libL1TriggerL1TTrackMatch.so+0x30e22)
    #3 0x146ba5805728 in L1TrackerEtMissEmulatorProducer::L1TrackerEtMissEmulatorProducer(edm::ParameterSet const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02788/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-06-09-2300/lib/el8_amd64_gcc11/pluginL1TrackTriggerPlugins.so+0x190728)
    #4 0x146ba586c06d in edm::stream::ProducingModuleAdaptor<L1TrackerEtMissEmulatorProducer, edm::stream::EDProducerBase, edm::stream::EDProducerAdaptorBase>::setupStreamModules() (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02788/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-06-09-2300/lib/el8_amd64_gcc11/pluginL1TrackTriggerPlugins.so+0x1f706d)
    #5 0x146c3a1fc9e8 in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doPreallocate(edm::PreallocationConfiguration const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02788/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-06-09-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x79a9e8)
    #6 0x146c3a18aaa6 in edm::Maker::makeModule(edm::MakeModuleParams const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02788/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-06-09-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x728aa6)
    #7 0x146c39ec2fc9 in edm::Factory::makeModule(edm::MakeModuleParams const&, edm::ModuleTypeResolverMaker const*, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02788/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-06-09-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x460fc9)
    #8 0x146c39f05b18 in edm::ModuleRegistry::getModule(edm::MakeModuleParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02788/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-06-09-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x4a3b18)
    #9 0x146c3a17d4bd in edm::WorkerRegistry::getWorker(edm::WorkerParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02788/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-06-09-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x71b4bd)
    #10 0x146c3a17e6cd in edm::WorkerManager::getWorker(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02788/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-06-09-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x71c6cd)
    #11 0x146c3a17eed9 in edm::WorkerManager::addToUnscheduledWorkers(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02788/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-06-09-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x71ced9)
    #12 0x146c3a0ddecc in edm::StreamSchedule::StreamSchedule(std::shared_ptr<edm::TriggerResultInserter>, std::vector<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> > > >&, std::vector<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> > > >&, std::shared_ptr<edm::ModuleRegistry>, edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::PreallocationConfiguration const&, edm::ProductRegistry&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration const>, edm::StreamID, edm::ProcessContext const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02788/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-06-09-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x67becc)
    #13 0x146c3a062987 in edm::Schedule::Schedule(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::ProductRegistry&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration const>, edm::PreallocationConfiguration const&, edm::ProcessContext const*, edm::ModuleTypeResolverMaker const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02788/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-06-09-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x600987)
    #14 0x146c3a0d0a72 in edm::ScheduleItems::initModules(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::PreallocationConfiguration const&, edm::ProcessContext const*, edm::ModuleTypeResolverMaker const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02788/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-06-09-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x66ea72)
    #15 0x146c39e10cb1 in tbb::detail::d1::function_task<edm::EventProcessor::init(std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) [clone .lto_priv.0] (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02788/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-06-09-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x3aecb1)
    #16 0x146c37ced303 in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/task_dispatcher.h:322
    #17 0x146c37ced303 in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/task_dispatcher.h:458
    #18 0x146c37ced303 in tbb::detail::r1::arena::process(tbb::detail::r1::thread_data&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/arena.cpp:137
    #19 0x146c37ced303 in tbb::detail::r1::market::process(rml::job&) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/market.cpp:599
    #20 0x146c37cef4c5 in tbb::detail::r1::rml::private_worker::run() /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:271
    #21 0x146c37cef4c5 in tbb::detail::r1::rml::private_worker::thread_routine(void*) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:221

Thread T2 created by T0 here:
    #0 0x146c3a510716 in __interceptor_pthread_create ../../../../libsanitizer/asan/asan_interceptors.cpp:216
    #1 0x146c37ceeb9a in tbb::detail::r1::rml::internal::thread_monitor::launch(void* (*)(void*), void*, unsigned long) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/rml_thread_monitor.h:195
    #2 0x146c37ceeb9a in tbb::detail::r1::rml::private_worker::wake_or_launch() /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:305
    #3 0x146c37ceeb9a in tbb::detail::r1::rml::private_server::wake_some(int) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:412
    #4 0x60c000441f3f  (<unknown module>)

Thread T4 created by T0 here:
    #0 0x146c3a510716 in __interceptor_pthread_create ../../../../libsanitizer/asan/asan_interceptors.cpp:216
    #1 0x146c37ceeb9a in tbb::detail::r1::rml::internal::thread_monitor::launch(void* (*)(void*), void*, unsigned long) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/rml_thread_monitor.h:195
    #2 0x146c37ceeb9a in tbb::detail::r1::rml::private_worker::wake_or_launch() /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:305
    #3 0x146c37ceeb9a in tbb::detail::r1::rml::private_server::wake_some(int) /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_13_1_0_pre3-el8_amd64_gcc11/build/CMSSW_13_1_0_pre3-build/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-4e779f195a25a0aba119b27519937ba0/tbb-v2021.8.0/src/tbb/private_server.cpp:412
    #4 0x146c3193007f  (<unknown module>)

SUMMARY: AddressSanitizer: heap-buffer-overflow (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02788/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_ASAN_X_2023-06-09-2300/lib/el8_amd64_gcc11/pluginL1TrackTriggerPlugins.so+0x1a76a9) in ap_fixed_base<20, 15, true, (ap_q_mode)4, (ap_o_mode)0, 0>::RType<10, 1, false>::mult ap_fixed_base<20, 15, true, (ap_q_mode)4, (ap_o_mode)0, 0>::operator*<10, 1, false, (ap_q_mode)4, (ap_o_mode)0, 0>(ap_fixed_base<10, 1, false, (ap_q_mode)4, (ap_o_mode)0, 0> const&) const
Shadow bytes around the buggy address:
  0x0c4a815a8bd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c4a815a8be0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c4a815a8bf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c4a815a8c00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c4a815a8c10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c4a815a8c20:[fa]fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a815a8c30: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a815a8c40: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a815a8c50: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a815a8c60: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a815a8c70: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==231397==ABORTING

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el8_amd64_gcc11/CMSSW_13_2_ASAN_X_2023-06-09-2300/pyRelValMatrixLogs/run/23834.999_TTbar_14TeV+2026D95PU_PMXS1S2PR/step3_TTbar_14TeV+2026D95PU_PMXS1S2PR.log#/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
l1-pending pending-signatures upgrade-pending
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@makortel @iarspider @cmsbuild @aloeliger @aandvalenzuela