[New Feature][ASH] Active history for sessions

[hbhanawat]

Description

For effective performance diagnosis, we need a mechanism which provides both current & historical view of system activity- such as what statements were running at time T, what were they doing at that time (on the CPU, waiting for locks, disk IO, network RPC, safe time, etc.), breakdown of that information by node or query shape (text), was a particular tablet/shard hotter than others, etc.. This mechanism should be on-by-default -- so it must be light-weight enough.

This issue tracks DB side changes needed for active history.

1. PG infrastructure for ASH
   • ([YSQL][ActiveHistory] Create YB AUH extension with circular buffer #19127)
   • ([YSQL][ASH] Refactor PG Active Session History code #20180)
   • ([YSQL][ActiveHistory] Collect wait events from PG #19129)
   • ([YSQL][ASH] Collect wait events from TServer #20240)
   • ([YSQL][ASH] Convert wait state code to wait event component, wait event class and wait event  #19130)
   • ([YSQL][ASH] Create a function and view yb_wait_event_desc #19131)
   • ([YSQL][ASH] Function and view for yb_tablet_info #19132)
   • ([YSQL][ASH] Function and view for yb_table_info #19133)
   • ([YSQL][ASH] Circular buffer should be based on the cores available #19134)
   • ([YSQL][ActiveHistory] Rename Active Universe History to Active Session History #19948)
   • ([YSQL][ASH] Create a Yb_active_universe_history function  #19128)
   • ([YSQL][ActiveHistory] Background workers with postgres backends can crash due to an assertion failure in ASH #20338)
   • ([YSQL][ActiveHistory] Add additional checks when ASH is enabled by default #20362)
   • ([YSQL][ActiveHistory] PgMiniAsh test failure in tsan builds #20387)
   • ([YSQL][ASH] Flaky test 'TestYbAsh#testEmptyCircularBuffer' #20629)
   • ([YSQL][ASH] Update ASH function names to pascal case #20677)
   • ([YSQL][ASH] Rename session id and tserver endpoint columns  #20920)
   • ([YSQL][ASH] Gflag fixes for Active session history #20922)
   • ([YSQL][ASH]Fix PG pulling of tserver compaction events #20923)
   • ([YSQL][ASH] Dump output of some predefined views of ASH on :13000/ endpoint #20925)
   • ([YSQL][ASH] Env variable yb_enable_ash is missing if it's not set via tserver flags #20996)
   • ([YSQL][ASH] New wait event required for the background worker ASH collector #20997)
   • ([YSQL][ASH] Pggate test failures when ASH is turned on by default #21010)
2. PG Instrumentation
   • ([YSQL][ActiveHistory] Instrument the query paths so that the AUH metadata is generated and is passed to tserver #19135)
   • ([YSQL][ASH] Active Session History Infrastructure for PGGate layer #20022)
   • ([YSQL][ASH] Generate query ID for utility statements (DDLs) #19136)
   • ([YSQL][ASH] Instrumentation of PGGate Calls #19137)
   • ([YSQL][ASH] Add session ID to the ASH metadata and view #20242)
   • ([YSQL][ASH] Sample catalog requests in ASH #20676)
   • ([YSQL][ASH] YSQL samples have incorrect wait event aux #21009)
   • ([YSQL][ASH] CREATE EXTENSION script fails when ASH is enabled #21030)
   • ([YSQL][ASH] Nested statements should be tracked only if pg_stat_statements tracks them #21031)
3. TServer - Wait State Infrastructure
   • ([DocDB][ActiveHistory] Macro additions and wait states declaration for AUH #19138)
   • ([DocDB][ASH] Random number generator for AUH #19149)
   • ([DocDB][ASH] Ability to push/pop wait states #19150)
   • ([DocDB][ActiveHistory] RPC call to fetch the AUH metadata from TServer #19139)
   • ([DocDB][ASH] Ability to Freeze WaitState in time while fetching waitstates. #19140)
   • ([DocDB][ASH] Support conditional setting of wait-states for AUH #19141)
   • ([DocDB][ASH] Thread restrictions for wait state transition #19142)
   • ([DocDB][ASH] Sizes of various ASH metadata fields are inconsistent #20454)
   • ([DocDB][ASH] Fix the wait state events, class and component names #20921)
4. TServer - Instrumentation
   • ([DocDB][ActiveHistory] Instrument Read/Write code paths #19143)
   • ([YCQL][ASH] CQL Instrumentation #19144)
   • ([DocDB][ASH] Add instrumentation infrastructure for compaction and flushes #19145)
   • ([DocDB][ASH] Instrument compaction and flushes #19146)
   • ([DocDB][ActiveHistory]Track calls to local tserver in the list of RPCs  #19750)
   • ([DocDB][ASH] Index Backfill instrumentation for ASH #20241)
   • ([DocDB][ASH] YBClient wait events are needed for YCQL but not for YSQL #20868)
   • ([DocDB][ASH] Instrument remote and local bootstrap with wait events #23198)
   • ([DocDB][ASH] Instrument CDC with wait events #23199)
   • ([DocDB][ASH] Instrument backup and snapshotting with wait events #23200)
   • ([DocDB][ASH] Instrument restore with wait events #23201)
   • ([DocDB][ASH] Instrument XCluster with wait events #23202)
5. Master - Instrumentation

• ([DocDB] Ash: Support Ash calls to the master #20671)

Warning: Please confirm that this issue does not contain any sensitive information

• I confirm this issue does not contain any sensitive information.