Remove YBClient from postgres #9936
Assignees
Labels
kind/enhancement
This is an enhancement of an existing feature
Comments
spolitov
added a commit
that referenced
this issue
Sep 14, 2021
Summary: This diff adds PgClient and PgClientService. And implements all required infrastructure to use it from postgres. Test Plan: Jenkins Reviewers: dmitry Reviewed By: dmitry Subscribers: ybase, bogdan Differential Revision: https://phabricator.dev.yugabyte.com/D12910
spolitov
added a commit
that referenced
this issue
Sep 19, 2021
Summary: This diff continue migration to PgClient and introduces the following function with their usage: 1) OpenTable 2) GetDatabaseInfo 3) IsInitDbDone Test Plan: Jenkins Reviewers: dmitry Reviewed By: dmitry Subscribers: ybase, bogdan Differential Revision: https://phabricator.dev.yugabyte.com/D13034
spolitov
added a commit
that referenced
this issue
Sep 24, 2021
Summary: After 442fe79/D13034 ysql_dump started to query shared data from master leader. But it resolves master addresses to endpoints before doing this query. That does not work in encrypted environment, since information about master host address is lost. Fixed by switching to original hostnames. Test Plan: Jenkins Reviewers: sanketh, dmitry Reviewed By: dmitry Subscribers: ybase Differential Revision: https://phabricator.dev.yugabyte.com/D13175
spolitov
added a commit
that referenced
this issue
Sep 27, 2021
Summary: This diff continue migration to PgClient and introduces the following function with their usage: 1) AlterTable 2) CreateTable 3) CreateDatabase Test Plan: Jenkins Reviewers: dmitry Reviewed By: dmitry Subscribers: bogdan, ybase Differential Revision: https://phabricator.dev.yugabyte.com/D13133
spolitov
added a commit
that referenced
this issue
Sep 28, 2021
Summary: Moved the following functionality: - AlterDatabase - BackfillIndex - CreateSequencesDataTable - CreateTablegroup - DropDatabase - DropTable - DropTablegroup - GetCatalogMasterVersion - GetDatabaseInfo - ListLiveTabletServers - TabletServerCount - TruncateTable Test Plan: Jenkins Reviewers: dmitry Reviewed By: dmitry Subscribers: bogdan, ybase Differential Revision: https://phabricator.dev.yugabyte.com/D13219
spolitov
added a commit
that referenced
this issue
Oct 1, 2021
Summary: This diff fixes ysql_dump issue in encrypted k8s environment by correct hostname usage. Test Plan: Launch encrypted k8s universe. Create table. Backup this table. Reviewers: dmitry Reviewed By: dmitry Subscribers: jenkins-bot, ybase, bogdan Differential Revision: https://phabricator.dev.yugabyte.com/D13271
spolitov
added a commit
that referenced
this issue
Oct 8, 2021
Summary: This diff changes logic in generating pg client session id. Now it is generated in tserver side. Test Plan: Jenkins Reviewers: dmitry Reviewed By: dmitry Subscribers: bogdan, ybase Differential Revision: https://phabricator.dev.yugabyte.com/D13339
spolitov
added a commit
that referenced
this issue
Oct 13, 2021
Summary: This diff fixes ysql_dump issue in encrypted k8s environment by correct hostname usage. Test Plan: Launch encrypted k8s universe. Create table. Backup this table. Jenkins: rebase: 2.9.1 Original commit: 83a6ed6/D13271 Reviewers: dmitry Reviewed By: dmitry Subscribers: bogdan, ybase, jenkins-bot Differential Revision: https://phabricator.dev.yugabyte.com/D13388
spolitov
added a commit
that referenced
this issue
Oct 21, 2021
Summary: Currently, batcher tries to look up the tablet as soon as the operation was added to it. This logic is not necessary for our scenarios since we flush batcher as soon as it was filled. Also in most cases, the meta cache already contains a tablet for the operation, so lookup completes in the same thread. Simplified batcher based on this logic - removed superfluous mutex and introduced plain steps in batcher flush. Also stopped creating batcher on `YBSession::SetTransaction`, so it is created only on `Apply`. Test Plan: Jenkins Reviewers: timur Reviewed By: timur Subscribers: bogdan, ybase Differential Revision: https://phabricator.dev.yugabyte.com/D13411
spolitov
added a commit
that referenced
this issue
Oct 28, 2021
Summary: Each RPC call contains RequestHeader protobuf, that is parsed into appropriate class. This protobuf contains 2 strings in remote_method field, where first string (service name) usually quite long, so memory is allocated for this string. This diff introduces simplified class for RequestHeader protobuf, so only call_id and timeout_ms are being parsed. While only bounds are determined for remote_method and slice is used to point to it. Test Plan: Jenkins Reviewers: dmitry Reviewed By: dmitry Subscribers: ybase, bogdan Differential Revision: https://phabricator.dev.yugabyte.com/D13470
jasonyb
pushed a commit
that referenced
this issue
Nov 23, 2021
Summary:
On CREATE INDEX using online schema change (default), one of the steps
is issuing the BackfillIndex request. Previously, postgres sent this
directly to master, and the timeout was determined by
backfill_index_client_rpc_timeout_ms (default 60m):
60m
postgres ---------> master
Due to commit 99e2395 titled
[#9936] Remove all direct YBClient usage from PgSession
this changed to
2m 60m
postgres ---------> tserver ---------> master
because the postgres-tserver RPC uses
yb_client_admin_operation_timeout_sec (default 2m) like the others.
This effectively makes the 60m timeout useless since the 2m one gets hit
first, and it easily makes long-running backfills time out.
Fix by using backfill_index_client_rpc_timeout_ms in the
postgres-tserver RPC. Then, by default, it should be
60m 60m
postgres ---------> tserver ---------> master
Close: #10646
Test Plan:
#!/usr/bin/env bash
set -e
buildtype=${1:-fastdebug}
for test in DropAfterFail \
DropWhileBackfilling \
LowerDefaultClientTimeout \
MasterLeaderStepdown \
WaitBackfillTimeout; do
./yb_build.sh $buildtype \
--cxx-test pgwrapper_pg_index_backfill-test \
--gtest_filter PgIndexBackfillTest."$test"
done
Reviewers: dpatra, zyu
Reviewed By: zyu
Subscribers: yql
Differential Revision: https://phabricator.dev.yugabyte.com/D14055
jasonyb
pushed a commit
that referenced
this issue
Nov 23, 2021
Summary:
On CREATE INDEX using online schema change (default), one of the steps
is issuing the BackfillIndex request. Previously, postgres sent this
directly to master, and the timeout was determined by
backfill_index_client_rpc_timeout_ms (default 60m):
60m
postgres ---------> master
Due to commit 99e2395 titled
[#9936] Remove all direct YBClient usage from PgSession
this changed to
2m 60m
postgres ---------> tserver ---------> master
because the postgres-tserver RPC uses
yb_client_admin_operation_timeout_sec (default 2m) like the others.
This effectively makes the 60m timeout useless since the 2m one gets hit
first, and it easily makes long-running backfills time out.
Fix by using backfill_index_client_rpc_timeout_ms in the
postgres-tserver RPC. Then, by default, it should be
60m 60m
postgres ---------> tserver ---------> master
Original Commit: 502ca3f
Original Differential Revision: https://phabricator.dev.yugabyte.com/D14055
Test Plan:
#!/usr/bin/env bash
set -e
buildtype=${1:-fastdebug}
for test in DropAfterFail \
DropWhileBackfilling \
LowerDefaultClientTimeout \
MasterLeaderStepdown \
WaitBackfillTimeout; do
./yb_build.sh $buildtype \
--cxx-test pgwrapper_pg_index_backfill-test \
--gtest_filter PgIndexBackfillTest."$test"
done
Reviewers: zyu
Reviewed By: zyu
Differential Revision: https://phabricator.dev.yugabyte.com/D14082
spolitov
added a commit
that referenced
this issue
Mar 4, 2022
Summary: Postgres layer uses `YBClient` to perform read and write operations. Since Postgres creates a separate process for each incoming connection, it creates `YBClient` for each incoming connection. But `YBClient` establishes connections to all tservers in the cluster and to the master leader. As result in the case of large systems with a lot of incoming connections, we get a lot of connections between nodes. It consumes resources and slows down the whole system. This diff moves read and write logic from postgres layer to the previously designed PgClient. That establishes a single local connection from postgres to tserver. Test Plan: Jenkins Reviewers: dmitry Reviewed By: dmitry Subscribers: jason, yql, ybase, bogdan Differential Revision: https://phabricator.dev.yugabyte.com/D13244
jayant07-yb
pushed a commit
to jayant07-yb/yugabyte-db
that referenced
this issue
Mar 8, 2022
Summary: Postgres layer uses `YBClient` to perform read and write operations. Since Postgres creates a separate process for each incoming connection, it creates `YBClient` for each incoming connection. But `YBClient` establishes connections to all tservers in the cluster and to the master leader. As result in the case of large systems with a lot of incoming connections, we get a lot of connections between nodes. It consumes resources and slows down the whole system. This diff moves read and write logic from postgres layer to the previously designed PgClient. That establishes a single local connection from postgres to tserver. Test Plan: Jenkins Reviewers: dmitry Reviewed By: dmitry Subscribers: jason, yql, ybase, bogdan Differential Revision: https://phabricator.dev.yugabyte.com/D13244
spolitov
added a commit
that referenced
this issue
Mar 11, 2022
Summary: This diff moves operations on sequences and removes YBClient instance from postgres. Test Plan: Jenkins Reviewers: dmitry Reviewed By: dmitry Subscribers: yql, ybase, bogdan Differential Revision: https://phabricator.dev.yugabyte.com/D15831
spolitov
added a commit
that referenced
this issue
Mar 15, 2022
Summary: This diff addresses remaining TODO for PgClient - remove dependency on YBTable in PgTableDesc. Also fixed issue with header guards in generated yrpc files. Test Plan: Jenkins Reviewers: dmitry Reviewed By: dmitry Subscribers: ybase Differential Revision: https://phabricator.dev.yugabyte.com/D15952
spolitov
added a commit
that referenced
this issue
Mar 15, 2022
Summary: Increase PgClient queue length from 50 to 5000. Configurable with gflag - pg_client_svc_queue_length. Test Plan: Jenkins Reviewers: bogdan Reviewed By: bogdan Subscribers: rahuldesirazu, jhe, ybase Differential Revision: https://phabricator.dev.yugabyte.com/D15995
spolitov
added a commit
that referenced
this issue
Mar 17, 2022
Summary: The deadline was not propagated to GetMetadata call from SetupSession. And default timeout was used for this operation, i.e. 5 seconds. Since this test creates local transaction status tables. It could take a longer time to pick status tablet, since it waits for all transaction status tables to be created. And statement was failing because of timeout in suboperation. While whole operation timeout was not yet reached. Fixed by propagating actual deadline to GetMetadata and Take functions. Also fixed GetTransactionStatusTablets to avoid holding shared catalog manager lock while waiting for status table to be created. Test Plan: ybd --java-test org.yb.pgsql.TestTransactionStatusTable#testCreation -n 20 Reviewers: esheng Reviewed By: esheng Subscribers: ybase, bogdan Differential Revision: https://phabricator.dev.yugabyte.com/D15981
d-uspenskiy
added a commit
that referenced
this issue
Jul 11, 2022
Summary: YSQL uses PgClientService for communications with remote masters/t-servers. Lots of the classes from ybc_client are not used anymore. This diff removes unused code/includes/dependencies. Test Plan: Jenkins Reviewers: sergei, mihnea, alex Reviewed By: sergei, alex Subscribers: yql Differential Revision: https://phabricator.dev.yugabyte.com/D18145
spolitov
added a commit
that referenced
this issue
Sep 1, 2022
Summary: Increase PgClient queue length from 50 to 5000. Configurable with gflag - pg_client_svc_queue_length. Original commit: c1e43ee/D15995 Test Plan: Jenkins Reviewers: bogdan Reviewed By: bogdan Subscribers: ybase, jhe, rahuldesirazu Differential Revision: https://phabricator.dev.yugabyte.com/D19296
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
No description provided.
The text was updated successfully, but these errors were encountered: