Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[YSQL] [Index Backfill] TestIndexBackfill failing #5044

Closed
frozenspider opened this issue Jul 10, 2020 · 1 comment
Closed

[YSQL] [Index Backfill] TestIndexBackfill failing #5044

frozenspider opened this issue Jul 10, 2020 · 1 comment
Assignees
@frozenspider
Labels
area/ysql Yugabyte SQL (YSQL) kind/failing-test Tests and testing infra
Milestone
v2.2

Comments

@frozenspider
Copy link
Contributor

frozenspider commented Jul 10, 2020 •
edited
Loading

(Test was enabled in #4941)
This test fails in release (due to error message change in 8246ca4#diff-35b7aae825e891e35b810e7bc838ffe1L159), as well as in ASAN
@frozenspider frozenspider added area/ysql Yugabyte SQL (YSQL) kind/failing-test Tests and testing infra labels Jul 10, 2020
@frozenspider frozenspider self-assigned this Jul 10, 2020
@frozenspider
Copy link
Contributor Author

ASAN was unhappy about debug_query_string pointing to destroyed MC when invoking elog:

ts1|pid6250|:15749|http://127.40.61.39:10878 ==6767==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x619000117580 at pc 0x0000014e920a bp 0x7ffea1d95220 sp 0x7ffea1d95218
ts1|pid6250|:15749|http://127.40.61.39:10878 READ of size 1 at 0x619000117580 thread T0
ts1|pid6250|:15749|http://127.40.61.39:10878     #0 0x14e9209 in append_with_tabs /home/centos/code/yugabyte/src/postgres/src/backend/utils/error/../../../../../../../src/postgres/src/backend/utils/error/elog.c:3755:15
ts1|pid6250|:15749|http://127.40.61.39:10878     #1 0x14e4576 in send_message_to_server_log /home/centos/code/yugabyte/src/postgres/src/backend/utils/error/../../../../../../../src/postgres/src/backend/utils/error/elog.c:3057:3
ts1|pid6250|:15749|http://127.40.61.39:10878     #2 0x14de516 in EmitErrorReport /home/centos/code/yugabyte/src/postgres/src/backend/utils/error/../../../../../../../src/postgres/src/backend/utils/error/elog.c:1578:3
ts1|pid6250|:15749|http://127.40.61.39:10878     #3 0x14dda92 in errfinish /home/centos/code/yugabyte/src/postgres/src/backend/utils/error/../../../../../../../src/postgres/src/backend/utils/error/elog.c:516:2
ts1|pid6250|:15749|http://127.40.61.39:10878     #4 0x111e668 in YBPrepareCacheRefreshIfNeeded /home/centos/code/yugabyte/src/postgres/src/backend/tcop/../../../../../../src/postgres/src/backend/tcop/postgres.c:3824:5
ts1|pid6250|:15749|http://127.40.61.39:10878     #5 0x111c7e0 in PostgresMain /home/centos/code/yugabyte/src/postgres/src/backend/tcop/../../../../../../src/postgres/src/backend/tcop/postgres.c:5006:7
ts1|pid6250|:15749|http://127.40.61.39:10878     #6 0xf6130f in BackendRun /home/centos/code/yugabyte/src/postgres/src/backend/postmaster/../../../../../../src/postgres/src/backend/postmaster/postmaster.c:4428:2
ts1|pid6250|:15749|http://127.40.61.39:10878     #7 0xf60056 in BackendStartup /home/centos/code/yugabyte/src/postgres/src/backend/postmaster/../../../../../../src/postgres/src/backend/postmaster/postmaster.c:4094:3
ts1|pid6250|:15749|http://127.40.61.39:10878     #8 0xf5e190 in ServerLoop /home/centos/code/yugabyte/src/postgres/src/backend/postmaster/../../../../../../src/postgres/src/backend/postmaster/postmaster.c:1737:7
ts1|pid6250|:15749|http://127.40.61.39:10878     #9 0xf59a89 in PostmasterMain /home/centos/code/yugabyte/src/postgres/src/backend/postmaster/../../../../../../src/postgres/src/backend/postmaster/postmaster.c:1400:11
ts1|pid6250|:15749|http://127.40.61.39:10878     #10 0xd3e8e3 in PostgresServerProcessMain /home/centos/code/yugabyte/src/postgres/src/backend/main/../../../../../../src/postgres/src/backend/main/main.c:234:3
ts1|pid6250|:15749|http://127.40.61.39:10878     #11 0xd3efd1 in main (/home/centos/code/yugabyte/build/asan-clang-dynamic/postgres/bin/postgres+0xd3efd1)
ts1|pid6250|:15749|http://127.40.61.39:10878     #12 0x7f3764094824 in __libc_start_main /tmp/glibc-20181130-26094-cs1x60/glibc-2.23/csu/../csu/libc-start.c:289
ts1|pid6250|:15749|http://127.40.61.39:10878     #13 0x495f08 in _start /tmp/glibc-20181130-26094-cs1x60/glibc-2.23/csu/../sysdeps/x86_64/start.S:118
ts1|pid6250|:15749|http://127.40.61.39:10878 
ts1|pid6250|:15749|http://127.40.61.39:10878 0x619000117580 is located 0 bytes to the right of 1024-byte region [0x619000117180,0x619000117580)
ts1|pid6250|:15749|http://127.40.61.39:10878 allocated by thread T0 here:
ts1|pid6250|:15749|http://127.40.61.39:10878     #0 0x564c6f in __interceptor_malloc /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20200625021759-2dafffee0f-centos/src/llvm-7.1.0.src/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:146
ts1|pid6250|:15749|http://127.40.61.39:10878     #1 0x15587ed in AllocSetContextCreateExtended /home/centos/code/yugabyte/src/postgres/src/backend/utils/mmgr/../../../../../../../src/postgres/src/backend/utils/mmgr/aset.c:477:19
ts1|pid6250|:15749|http://127.40.61.39:10878     #2 0x1496ac9 in BuildCachedPlan /home/centos/code/yugabyte/src/postgres/src/backend/utils/cache/../../../../../../../src/postgres/src/backend/utils/cache/plancache.c:957:18
ts1|pid6250|:15749|http://127.40.61.39:10878     #3 0x1494c81 in GetCachedPlan /home/centos/code/yugabyte/src/postgres/src/backend/utils/cache/../../../../../../../src/postgres/src/backend/utils/cache/plancache.c:1187:11
ts1|pid6250|:15749|http://127.40.61.39:10878     #4 0x111c8c4 in PostgresMain /home/centos/code/yugabyte/src/postgres/src/backend/tcop/../../../../../../src/postgres/src/backend/tcop/postgres.c:5038:28
ts1|pid6250|:15749|http://127.40.61.39:10878     #5 0xf6130f in BackendRun /home/centos/code/yugabyte/src/postgres/src/backend/postmaster/../../../../../../src/postgres/src/backend/postmaster/postmaster.c:4428:2
ts1|pid6250|:15749|http://127.40.61.39:10878     #6 0xf60056 in BackendStartup /home/centos/code/yugabyte/src/postgres/src/backend/postmaster/../../../../../../src/postgres/src/backend/postmaster/postmaster.c:4094:3
ts1|pid6250|:15749|http://127.40.61.39:10878     #7 0xf5e190 in ServerLoop /home/centos/code/yugabyte/src/postgres/src/backend/postmaster/../../../../../../src/postgres/src/backend/postmaster/postmaster.c:1737:7
ts1|pid6250|:15749|http://127.40.61.39:10878     #8 0xf59a89 in PostmasterMain /home/centos/code/yugabyte/src/postgres/src/backend/postmaster/../../../../../../src/postgres/src/backend/postmaster/postmaster.c:1400:11
ts1|pid6250|:15749|http://127.40.61.39:10878     #9 0xd3e8e3 in PostgresServerProcessMain /home/centos/code/yugabyte/src/postgres/src/backend/main/../../../../../../src/postgres/src/backend/main/main.c:234:3
ts1|pid6250|:15749|http://127.40.61.39:10878     #10 0xd3efd1 in main (/home/centos/code/yugabyte/build/asan-clang-dynamic/postgres/bin/postgres+0xd3efd1)
ts1|pid6250|:15749|http://127.40.61.39:10878     #11 0x7f3764094824 in __libc_start_main /tmp/glibc-20181130-26094-cs1x60/glibc-2.23/csu/../csu/libc-start.c:289
ts1|pid6250|:15749|http://127.40.61.39:10878

@frozenspider frozenspider added this to the v2.2 milestone Jul 10, 2020
frozenspider added a commit that referenced this issue Jul 10, 2020
@frozenspider
#5044: [YSQL] [Index Backfill] Fix failing TestIndexBackfill
4c4260d 
Summary:
`org.yb.pgsql.TestIndexBackfill` test has been failing for the following reasons:
* Error message it was expecting to catch was changed in 8246ca4#diff-35b7aae825e891e35b810e7bc838ffe1L159
* For "schema version mismatch" transparent restart, when aborting transaction, we weren't cleaning up `debug_query_string` which remained pointing to the query allocated in the destroyed memory context - briefly, but it was subsequently used when logging, and was caught by ASAN

It might still sometimes fail in ASAN though:
* Because of #5030
* Weird case when test succeeds but our test framework reports it as a failure, possibly because of stray `postgres` processes left over after the test
(Resolves #5044)

Test Plan:
ybd --java-test org.yb.pgsql.TestIndexBackfill
(and ASAN as well)

Reviewers: jason, mihnea

Reviewed By: mihnea

Subscribers: yql

Differential Revision: https://phabricator.dev.yugabyte.com/D8878

(cherry picked from commit 91733d4)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@frozenspider frozenspider

Labels
area/ysql Yugabyte SQL (YSQL) kind/failing-test Tests and testing infra
Projects
None yet
Milestone
v2.2
Development

No branches or pull requests
1 participant
@frozenspider