Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[YSQL] YBCDestroyPgGate called twice in PgMiniTest.WriteRetry #3236

Closed
mbautin opened this issue Dec 31, 2019 · 2 comments
Closed

[YSQL] YBCDestroyPgGate called twice in PgMiniTest.WriteRetry #3236

mbautin opened this issue Dec 31, 2019 · 2 comments
Assignees
@mbautin
Labels
area/docdb YugabyteDB core features area/ysql Yugabyte SQL (YSQL) kind/failing-test Tests and testing infra

Comments

@mbautin
Copy link
Contributor

mbautin commented Dec 31, 2019 

[ RUN      ] PgMiniTest.WriteRetry
The files belonging to this database system will be owned by user "mbautin".
This user must also own the server process.

In YugaByte DB, setting LC_COLLATE to C and all other locale settings to en_US.UTF-8 by default. Locale support will be enhanced as part of addressing https://github.com/YugaByte/yugabyte-db/issues/1557The database cluster will be initialized with locales
  COLLATE:  C
  CTYPE:    en_US.UTF-8
  MESSAGES: en_US.UTF-8
  MONETARY: en_US.UTF-8
  NUMERIC:  en_US.UTF-8
  TIME:     en_US.UTF-8
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

creating directory /private/tmp/ybtest-501/pg_mini-test.PgMiniTest.WriteRetry.1577768074092276-19178/minicluster-data/ts-0-root/pg_data ... ok
creating subdirectories ... ok
selecting default max_connections ... 300
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
syncing data to disk ... ok
2019-12-30 20:54:58.357 PST [19200] LOG:  YugaByte is ENABLED in PostgreSQL. Transactions are enabled.
2019-12-30 20:54:58.359 PST [19200] LOG:  listening on IPv4 address "127.0.0.2", port 27495
2019-12-30 20:54:58.373 PST [19201] LOG:  database system was shut down at 2019-12-30 20:54:58 PST
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1230 20:54:58.373962 206869952 webserver.cc:148] Starting webserver on 127.0.0.2:13000
I1230 20:54:58.374320 206869952 webserver.cc:153] Document root: /Users/mbautin/code/yugabyte-db5/www
I1230 20:54:58.375098 206869952 webserver.cc:240] Webserver started. Bound to: http://127.0.0.2:13000/
2019-12-30 20:54:58.378 PST [19200] LOG:  database system is ready to accept connections
I1230 20:54:58.438290 206869952 pggate.cc:105] Reset YSQL bind address to 0.0.0.0:5432
I1230 20:54:58.438943 206869952 mem_tracker.cc:249] MemTracker: hard memory limit is 8.000000 GB
I1230 20:54:58.439106 206869952 mem_tracker.cc:251] MemTracker: soft memory limit is 6.800000 GB
W1230 20:54:58.441483 206869952 hybrid_clock.cc:115] HybridClock initialized in local mode (OS X only). Not suitable for distributed clusters.
I1230 20:54:58.441751 37277696 async_initializer.cc:73] Starting to init ybclient
W1230 20:55:03.451520 37277696 net_util.cc:369] Time spent looking up canonical hostname for localhost mbmbp.local: real 5.004s	user 0.000s	sys 0.001s
W1230 20:55:03.451915 37277696 client-internal.cc:830] Failed to get hostname: Network error (yb/util/net/net_util.cc:376): Unable to lookup FQDN (mbmbp.local), getaddrinfo returned 8 (EAI_NONAME): Undefined error: 0 (system error 0)
I1230 20:55:03.457404 37277696 async_initializer.cc:77] Successfully built ybclient
I1230 20:55:05.269498 206869952 ybccmds.c:472] Creating Table postgres.public.t
I1230 20:55:05.269651 206869952 pg_ddl.cc:183] PgCreateTable: creating a transactional table: postgres.t
I1230 20:55:05.566455 206869952 table_creator.cc:270] Created table postgres.t of type PGSQL_TABLE_TYPE
I1230 20:55:05.992771 38350848 tablet_rpc.cc:343] Already present (yb/consensus/retryable_requests.cc:239): Failed Write(tablet: 995687a3326f4fbbb981cd08c2eeff0a, num_ops: 1, num_attempts: 5, txn: 00000000-0000-0000-0000-000000000000) to tablet 995687a3326f4fbbb981cd08c2eeff0a on tablet server { uuid: d8df31b1f8c4423e8f244bb9badd688d private: [host: "127.0.0.4" port: 22468] public: [host: "127.0.0.5.ip.yugabyte" port: 22468] cloud_info: placement_cloud: "cloud1" placement_region: "rack2" placement_zone: "zone" after 5 attempt(s): Duplicate request
2019-12-30 20:55:06.004 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:06.004 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (1)
2019-12-30 20:55:06.062 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:06.062 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (12)
2019-12-30 20:55:06.092 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:06.092 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (13)
2019-12-30 20:55:06.130 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:06.130 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (17)
2019-12-30 20:55:06.161 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:06.161 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (19)
2019-12-30 20:55:06.229 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:06.229 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (21)
2019-12-30 20:55:06.274 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:06.274 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (27)
2019-12-30 20:55:06.480 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:06.480 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (28)
2019-12-30 20:55:06.507 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:06.507 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (30)
2019-12-30 20:55:06.526 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:06.526 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (31)
2019-12-30 20:55:06.549 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:06.549 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (32)
2019-12-30 20:55:06.582 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:06.582 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (34)
2019-12-30 20:55:06.647 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:06.647 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (37)
2019-12-30 20:55:06.787 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:06.787 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (43)
2019-12-30 20:55:06.811 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:06.811 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (44)
2019-12-30 20:55:06.848 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:06.848 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (49)
2019-12-30 20:55:06.899 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:06.899 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (58)
2019-12-30 20:55:06.938 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:06.938 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (62)
I1230 20:55:07.009588 37814272 tablet_rpc.cc:343] Already present (yb/consensus/retryable_requests.cc:239): Failed Write(tablet: 995687a3326f4fbbb981cd08c2eeff0a, num_ops: 1, num_attempts: 9, txn: 00000000-0000-0000-0000-000000000000) to tablet 995687a3326f4fbbb981cd08c2eeff0a on tablet server { uuid: d8df31b1f8c4423e8f244bb9badd688d private: [host: "127.0.0.4" port: 22468] public: [host: "127.0.0.5.ip.yugabyte" port: 22468] cloud_info: placement_cloud: "cloud1" placement_region: "rack2" placement_zone: "zone" after 9 attempt(s): Duplicate request
2019-12-30 20:55:07.010 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:07.010 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (64)
2019-12-30 20:55:07.038 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:07.038 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (66)
2019-12-30 20:55:07.061 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:07.061 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (70)
2019-12-30 20:55:07.094 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:07.094 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (72)
2019-12-30 20:55:07.145 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:07.145 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (82)
2019-12-30 20:55:07.212 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:07.212 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (83)
2019-12-30 20:55:07.234 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:07.234 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (84)
2019-12-30 20:55:07.282 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:07.282 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (93)
2019-12-30 20:55:07.364 PST [19205] ERROR:  duplicate key value violates unique constraint "t_pkey"
2019-12-30 20:55:07.364 PST [19205] STATEMENT:  INSERT INTO t (key) VALUES (1)
2019-12-30 20:55:07.364 PST [19200] LOG:  received immediate shutdown request
2019-12-30 20:55:07.365 PST [19205] WARNING:  terminating connection because of crash of another server process
2019-12-30 20:55:07.365 PST [19205] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2019-12-30 20:55:07.365 PST [19205] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
F1230 20:55:07.365284 206869952 ybc_pggate.cc:76] void yb::pggate::YBCDestroyPgGate() can only be called once
Fatal failure details written to /.FATAL.details.2019-12-30T20_55_07.pid19205.txt
F20191230 20:55:07 ../../src/yb/yql/pggate/ybc_pggate.cc:76] void yb::pggate::YBCDestroyPgGate() can only be called once
    @        0x10e7b09d0  google::LogDestination::LogToSinks()
    @        0x10e7afa2e  google::LogMessage::SendToLog()
    @        0x10e7b03a5  google::LogMessage::Flush()
    @        0x10e7b527f  google::LogMessageFatal::~LogMessageFatal()
    @        0x10e7b1379  google::LogMessageFatal::~LogMessageFatal()
    @        0x10762867e  YBCDestroyPgGate
    @        0x106ee1424  YBOnPostgresBackendShutdown
    @        0x106dbdcbb  quickdie
    @     0x7fff6e844b5d  _sigtramp
    @                0x0  (unknown)
    @        0x10b4da71c  yb::ConditionVariable::TimedWait()
    @        0x10b4da6c6  yb::ConditionVariable::WaitUntil()
    @        0x10b4db25a  yb::CountDownLatch::WaitUntil()
    @        0x10b4db2c9  yb::CountDownLatch::WaitFor()
    @        0x10b68e404  yb::ThreadJoiner::Join()
    @        0x109fd8d56  yb::rpc::Reactor::Join()
    @        0x109f94990  yb::rpc::Messenger::Shutdown()
    @        0x10762f2c2  yb::pggate::PgApiImpl::~PgApiImpl()
    @        0x10762f41e  yb::pggate::PgApiImpl::~PgApiImpl()
    @        0x10762858d  YBCDestroyPgGate
    @        0x106ee1424  YBOnPostgresBackendShutdown
    @        0x106d99034  proc_exit
    @        0x106dbf7f6  PostgresMain
    @        0x106d434fa  BackendRun
    @        0x106d42d80  BackendStartup
    @        0x106d42332  ServerLoop
    @        0x106d40185  PostmasterMain
    @        0x106caac2e  PostgresServerProcessMain
    @        0x106caaf72  main
    @     0x7fff6e6593d5  start
    @               0x13  (unknown)

*** Check failure stack trace: ***
    @        0x10e7afcd3  google::LogMessage::SendToLog()
    @        0x10e7b03a5  google::LogMessage::Flush()
    @        0x10e7b527f  google::LogMessageFatal::~LogMessageFatal()
    @        0x10e7b1379  google::LogMessageFatal::~LogMessageFatal()
    @        0x10762867e  YBCDestroyPgGate
    @        0x106ee1424  YBOnPostgresBackendShutdown
    @        0x106dbdcbb  quickdie
    @     0x7fff6e844b5d  _sigtramp
    @                0x0  (unknown)
    @        0x10b4da71c  yb::ConditionVariable::TimedWait()
    @        0x10b4da6c6  yb::ConditionVariable::WaitUntil()
    @        0x10b4db25a  yb::CountDownLatch::WaitUntil()
    @        0x10b4db2c9  yb::CountDownLatch::WaitFor()
    @        0x10b68e404  yb::ThreadJoiner::Join()
    @        0x109fd8d56  yb::rpc::Reactor::Join()
    @        0x109f94990  yb::rpc::Messenger::Shutdown()
    @        0x10762f2c2  yb::pggate::PgApiImpl::~PgApiImpl()
    @        0x10762f41e  yb::pggate::PgApiImpl::~PgApiImpl()
    @        0x10762858d  YBCDestroyPgGate
    @        0x106ee1424  YBOnPostgresBackendShutdown
    @        0x106d99034  proc_exit
    @        0x106dbf7f6  PostgresMain
    @        0x106d434fa  BackendRun
    @        0x106d42d80  BackendStartup
    @        0x106d42332  ServerLoop
    @        0x106d40185  PostmasterMain
    @        0x106caac2e  PostgresServerProcessMain
    @        0x106caaf72  main
    @     0x7fff6e6593d5  start
    @               0x13  (unknown)
2019-12-30 20:55:07.377 PST [19200] LOG:  database system is shut down
@mbautin mbautin self-assigned this Dec 31, 2019
@bmatican
Copy link
Contributor

bmatican commented Feb 28, 2020 •
edited
Loading

Looking over our 600 builds, just for FATAL errors, seems like we're hitting this quite a lot:

cat fatals_600 | xargs grep -l 'void yb::pggate::YBCDestroyPgGate()' | wc -l
70

Example tests:

/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/607/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_SystemTableTxnTest.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/607/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_DropDBUpdateSysTablet.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/624/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/626/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_DropDBUpdateSysTablet.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/629/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_DropDBUpdateSysTablet.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/634/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_SystemTableTxnTest.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/635/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/638/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/639/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/640/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/642/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/643/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/645/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/646/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/648/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/651/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_SystemTableTxnTest.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/651/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/654/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/655/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_SystemTableTxnTest.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/656/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_DropDBUpdateSysTablet.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/656/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/658/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/659/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/660/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_DropDBUpdateSysTablet.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/661/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/667/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/670/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/675/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/676/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/677/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/683/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_SystemTableTxnTest.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/683/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/684/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_DropDBUpdateSysTablet.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/686/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_SystemTableTxnTest.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/686/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/687/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_DropDBUpdateSysTablet.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/688/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_SystemTableTxnTest.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/689/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-asan/builds/696/archive/build/asan-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-debug/builds/603/archive/build/debug-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_DropDBUpdateSysTablet.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-debug/builds/605/archive/build/debug-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-debug/builds/615/archive/build/debug-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_DropDBMarkDeleted.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-debug/builds/617/archive/build/debug-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-debug/builds/635/archive/build/debug-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-debug/builds/647/archive/build/debug-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-debug/builds/648/archive/build/debug-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-debug/builds/651/archive/build/debug-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-debug/builds/655/archive/build/debug-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-debug/builds/661/archive/build/debug-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_DropDBUpdateSysTablet.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-debug/builds/666/archive/build/debug-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-release/builds/628/archive/build/release-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_DropDBUpdateSysTablet.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-release/builds/643/archive/build/release-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_DropDBUpdateSysTablet.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-release/builds/657/archive/build/release-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_DropDBUpdateSysTablet.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-clang-release/builds/665/archive/build/release-clang-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-debug/builds/602/archive/build/debug-gcc-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_DropDBUpdateSysTablet.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-debug/builds/607/archive/build/debug-gcc-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_DropDBUpdateSysTablet.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-debug/builds/614/archive/build/debug-gcc-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-debug/builds/624/archive/build/debug-gcc-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-debug/builds/625/archive/build/debug-gcc-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-debug/builds/633/archive/build/debug-gcc-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_DropDBUpdateSysTablet.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-debug/builds/637/archive/build/debug-gcc-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-debug/builds/639/archive/build/debug-gcc-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-debug/builds/645/archive/build/debug-gcc-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-debug/builds/658/archive/build/debug-gcc-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/622/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/664/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/669/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/672/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_DropDBUpdateSysTablet.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/678/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_BulkCopyWithRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/681/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_mini-test/PgMiniTest_DropDBUpdateSysTablet.log

cc @d-uspenskiy

This also generates various use-after-free errors in ASAN:

    #0 0x14c2215 in GetMemoryChunkContext
    #1 0x14c4a91 in pfree
    #2 0x143b976 in errhint
    #3 0x1099f28 in quickdie
    #4 0x7fb2455fbb9f  (
    #5 0x7fb2455f800c in __pthread_cond_wait
    #6 0x7fb23f497189 in void boost::asio::detail::posix_event::wait<boost::asio::detail::conditionally_enabled_mutex::scoped_lock>(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&)
    #7 0x7fb23f49656e in boost::asio::detail::conditionally_enabled_event::wait(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&)
    #8 0x7fb23f494c8d in boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&)
    #9 0x7fb23f494885 in boost::asio::detail::scheduler::run(boost::system::error_code&)
    #10 0x7fb23f48cf1a in yb::rpc::IoThreadPool::Impl::Execute()

#0 0x7f2fd02b4033 in std::__1::unique_ptr<yb::client::YBClient::Data, std::__1::default_delete<yb::client::YBClient::Data> >::operator->() const
#1 0x7f2fd02b4033 in yb::client::YBClient::IsMultiMaster() const
#2 0x7f2fd031f4e4 in yb::Status yb::client::YBClient::Data::SyncLeaderMasterRpc<yb::master::GetYsqlCatalogConfigRequestPB, yb::master::GetYsqlCatalogConfigResponsePB>(std::__1::chrono::time_point<yb::CoarseMonoClock, std::__1::chrono::d
#3 0x7f2fd02a1e53 in yb::client::YBClient::GetYsqlCatalogMasterVersion(unsigned long*)
#4 0x7f2fd2a1a9f5 in yb::pggate::PgSession::GetCatalogMasterVersion(unsigned long*)
#5 0x7f2fd29fd49c in yb::pggate::PgApiImpl::GetCatalogMasterVersion(unsigned long*)
#6 0x7f2fd29e98d7 in YBCPgGetCatalogMasterVersion
#7 0x10a3105 in YBRefreshCache
#8 0x10a3fec in YBCheckSharedCatalogCacheVersion
#9 0x109cfbf in PostgresMain
#10 0xef516b in BackendRun
#11 0xef4074 in BackendStartup
#12 0xef24c0 in ServerLoop
#13 0xeee4fa in PostmasterMain
#14 0xcd8113 in PostgresServerProcessMain
#15 0xcd87b1 in main (

#0 0x7f39ab8e25a5 in std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::basic_string(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) (
#1 0x7f39bcf3e109 in yb::HostPort::HostPort(yb::HostPort const&)
#2 0x7f39bcf65847 in yb::client::YBClient::Data::leader_master_hostport() const
#3 0x7f39bcf9c435 in yb::Status yb::client::YBClient::Data::SyncLeaderMasterRpc<yb::master::GetYsqlCatalogConfigRequestPB, yb::master::GetYsqlCatalogConfigResponsePB>(std::__1::chrono::time_point<yb::CoarseMonoClock, std::__1::chrono::d
#4 0x7f39bcf1ee53 in yb::client::YBClient::GetYsqlCatalogMasterVersion(unsigned long*)
#5 0x7f39bf6979f5 in yb::pggate::PgSession::GetCatalogMasterVersion(unsigned long*)
#6 0x7f39bf67a49c in yb::pggate::PgApiImpl::GetCatalogMasterVersion(unsigned long*)
#7 0x7f39bf6668d7 in YBCPgGetCatalogMasterVersion
#8 0x10a3105 in YBRefreshCache
#9 0x10a3fec in YBCheckSharedCatalogCacheVersion
#10 0x109cfbf in PostgresMain
#11 0xef516b in BackendRun
#12 0xef4074 in BackendStartup
#13 0xef24c0 in ServerLoop
#14 0xeee4fa in PostmasterMain
#15 0xcd8113 in PostgresServerProcessMain
#16 0xcd87b1 in main (

@bmatican bmatican added area/docdb YugabyteDB core features area/ysql Yugabyte SQL (YSQL) kind/failing-test Tests and testing infra labels Feb 28, 2020
d-uspenskiy added a commit that referenced this issue Apr 3, 2020
@d-uspenskiy
#3236 [YSQL] Remove redundant call of YBOnPostgresBackendShutdown
a6007a7 
Summary:
postgres function `die()` prepares process exit, real exit will be performed in `proc_exit` function in main thread.
`proc_exit` has a call of YBOnPostgresBackendShutdown function, it should not be called in `die()` function.

Test Plan: Jenkins

Reviewers: mikhail, oleg

Reviewed By: oleg

Subscribers: bogdan, oleg, yql

Differential Revision: https://phabricator.dev.yugabyte.com/D8186
@mbautin
Copy link
Contributor Author

mbautin commented Jul 2, 2020

@d-uspenskiy: can this issue be closed now?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mbautin mbautin

Labels
area/docdb YugabyteDB core features area/ysql Yugabyte SQL (YSQL) kind/failing-test Tests and testing infra
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@bmatican @mbautin @d-uspenskiy