Skip to content

Commit

Permalink
feat(docs): update sharding docs (#340)
Browse files Browse the repository at this point in the history
* feat(docs): update sharding docs

Signed-off-by: wbtlb <[email protected]>

* chore(docs): remove unuse file

Signed-off-by: wbtlb <[email protected]>

Signed-off-by: wbtlb <[email protected]>
Co-authored-by: wbtlb <[email protected]>
  • Loading branch information
wbtlb and wbtlb authored Sep 26, 2022
1 parent f78672c commit eaa780d
Show file tree
Hide file tree
Showing 20 changed files with 1,565 additions and 19 deletions.
15 changes: 12 additions & 3 deletions docs/docs/Features/sharding.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,13 @@ sidebar_position: 6
```

### 名词解释

SQL 解析: 在分片中,请求到达 Pisa-Proxy 后会首先经过 SQL Parser,将 SQL 解析成 AST。

SQL 改写: 解析结束后,Pisa-Proxy 会根据分片规则对当前 SQL 语句进行改写,生成真实要执行的 SQL 语句。

SQL 路由: Pisa-Proxy 会根据分片规则,将改写好的 SQL 语句路由到后端对应数据源上执行 SQL 语句。

结果归并: SQL 被下推执行之后,Pisa-Proxy 会对查询结果进行归并,并最终返回给客户端。

### SQL 改写介绍
Expand All @@ -37,6 +41,11 @@ SELECT order_id FROM order.t_order WHERE order_id = 1;
```
SELECT order_id FROM order.t_order_00001 WHERE order_id = 1;
```
下图展示了数据查询过程
![](/img/sharding-select.png)

以数据插入为例,数据写入过程如下图:
![](/img/sharding-select.png)

**特别说明: SQL rewrite 在修改标识符计算实际表名时会自动根据分片规则添加表索引,索引规则位 表名_索引,索引位为5位表示。例如:`t_order` 表改写后为 `t_order_00000`。因此用户需要根据实际业务场景先创建好对应的表名**

Expand Down Expand Up @@ -81,9 +90,9 @@ SELECT COUNT(price) AS AVG_DERIVED_COUNT_00000, SUM(price) AS AVG_DERIVED_SUM_00
| 属性 | 值类型 | 是否依赖 | 默认值 | 含义 |
|-----|-------|---------|-------|-----|
| table_name |String|||分片表名|
| actual_datanodes| Vec\<String\>|||后端数据源|
|binding_tables|Vec\<String\>|||暂不支持||
|broadcast_tables|Vec\<String\>|||暂不支持||
| actual_datanodes| array[String]|||后端数据源|
|binding_tables|arrayString]|||暂不支持||
|broadcast_tables|array[String]|||暂不支持||
|table_sharding_algorithm_name|enum|||分片算法|
|table_sharding_column|String|||分片键|
|sharding_count|u64|||分片数|
52 changes: 37 additions & 15 deletions docs/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion docs/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
"prism-react-renderer": "^1.3.1",
"react": "^17.0.2",
"react-dom": "^17.0.2",
"react-github-btn": "^1.2.1"
"react-github-btn": "^1.4.0"
},
"browserslist": {
"production": [
Expand Down
Binary file added docs/static/img/sharding-insert.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/static/img/sharding-select.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 4 additions & 0 deletions docs/versioned_docs/version-v0.3.0/Features/_category_.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"label": "特性",
"position": 3
}
51 changes: 51 additions & 0 deletions docs/versioned_docs/version-v0.3.0/Features/loadbalancer.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
---
sidebar_position: 1
---

# 负载均衡

负载均衡模块为 Pisa-Proxy 代理后端节点时,新建链接时对后端节点选取策略的实现。

## 已支持负载均衡策略
- [x] Random
- [x] RoundRobin

## 配置示例

若需要在代理中启用负载均衡,则需要配置 `simple_loadbalance` 块.
```
[proxy.config.simple_loadbalance]
balance_type = "roundrobin"
nodes = ["ds001","ds002"]
```
其中 balance_type: 可选值为 random,roundrobin,默认值为 random;nodes: 选取后端数据源中定义的 `name`.

后端数据源配置:
```
[[mysql.node]]
name = "ds001"
user = "root"
password = "12345678"
db = "test"
host = "127.0.0.1"
port = 3306
[[mysql.node]]
name = "ds002"
user = "root"
password = "12345678"
db = "test"
host = "127.0.0.1"
port = 3307
```

## 模块设计

该模块定义了一个负载均衡的 Trait,封装了在 Pisa-Proxy 中对于负载均衡的构建方式。以及定义了 Random 和 RoundRobin 两种负载均衡策略所要实现的具体方法。

### 代码结构
FILES IN THIS DIRECTORY (loadbalance/src)
balance.rs - 负载均衡 Trait,定义了负载均衡方法和构建负载均衡模块
random_weighted.rs - random weighted 负载均衡策略
roundrobin_weighted.rs - roundrobin weighted 负载均衡策略

51 changes: 51 additions & 0 deletions docs/versioned_docs/version-v0.3.0/Features/mysql-protocol.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
---
sidebar_position: 2
---
# MySQL 协议

此功能主要为 Pisa-Proxy MySQL 代理的核心组件, 依据 [MySQL 协议](https://dev.mysql.com/doc/internals/en/client-server-protocol.html) 进行实现。该功能主要使用 Rust 生态的 [Tokio](https://github.com/tokio-rs/tokio) 异步运行时框架。其中对网络数据包的读写、协议的编码等操作都通过 Tokio 提供的工具集实现。


## 已支持命令
- [x] COM_INIT_DB
- [x] COM_QUERY
- [x] COM_FIELD_LIST
- [x] COM_QUIT
- [x] COM_PING
- [x] COM_STMT_PREPARE
- [x] COM_STMT_EXECUTE
- [x] COM_STMT_CLOSE
- [x] COM_STMT_RESET

## 支持认证方式
- [x] mysql_native_password
- [ ] sha256_password
- [ ] caching_sha2_password

## 设计说明

本模块共由3部分组成,对应 Pisa-Proxy 中作为客户端和服务端两部分。在 `server` 目录中主要定义了 Pisa-Proxy 作为服务端对客户端请求的处理逻辑。也包含了在 TCP 层的网络数据包的读写操作。在 `client` 目录中定义了 Pisa-Proxy 作为客户端对 MySQL 数据库的建立链接、握手认证和发送客户端命令等操作。

### 代码结构

FILES IN THIS DIRECTORY (protocol/mysql/src/client)
auth.rs - 对 MySQL 登陆认证
codec.rs - 对 MySQL 协议的解码
conn.rs - 对 MySQL 发起链接、握手及命令处理
err.rs - MySQL Error 类型定义
resultset.rs - 一些 MySQL ResultSet 结果处理方法
stmt.rs - 对 MySQL Prepare Statement 处理方法
stream.rs - 对 Tokio TcpSteam 封装

FILES IN THIS DIRECTORY (protocol/mysql/src/server)
conn.rs - 和客户端建链、握手认证处理
packet.rs - 网络数据包的读写操作
stream.rs - 对底层 Tcp 流的封装
tls.rs - MySQL TLS 链接封装

FILES IN THIS DIRECTORY (protocol/mysql/src)
charset.rs - MySQL 字符集声明
err.rs - 协议解析错误处理定义
mysql_const.rs。 - 常量定义
util.rs - 一些通用函数的实现

18 changes: 18 additions & 0 deletions docs/versioned_docs/version-v0.3.0/Features/observability.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
---
sidebar_position: 5
---

# 可观测性

Pisanix 目前在 Pisa-Proxy 处理 SQL 命令的时候采集相关指标,并以 `/metrics` 路径进行暴露。

## 已支持指标
- SQL_PROCESSED_TOTAL: 统计所有执行的 SQL 数量
- SQL_PROCESSED_DURATION: 统计所有 SQL 的执行时间
- SQL_UNDER_PROCESSING: 记录当前正在执行的 SQL 数量

测试效果如下图:

![grafana](/img/grafana.jpg)

下一步将支持更多标签和指标,如 SQL 语句类型、延迟、错误率、TopK、运行时资源等。
Loading

0 comments on commit eaa780d

Please sign in to comment.