Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

misc: merge main #581

Merged
merged 109 commits into from
Jan 16, 2023
Merged
Changes from 1 commit
Commits
Show all changes
109 commits
Select commit Hold shift + click to select a range
cf1b30c
feat(blog): add missing sqlx migration code to auth blog post (#408)
oddgrd Oct 18, 2022
ac43016
misc: 0.7.0 (#407)
chesedo Oct 19, 2022
37755ca
fix(cargo-shuttle): prevent crash when config owned by root (#409)
akrantz01 Oct 19, 2022
76f4fee
fix: use correct timeout start point (#410)
brokad Oct 20, 2022
607c3e1
feat(deployer): implement container memory limits (#411)
oddgrd Oct 20, 2022
e676715
bug: `transport error` when trying to connect to provisioner (#416)
chesedo Oct 20, 2022
6e771c7
feat: gateway admin revive (#412)
brokad Oct 20, 2022
8f7a341
bug: timeout curl health check on deployer (#415)
chesedo Oct 20, 2022
b4149f2
Article/beta article (#420)
Oct 21, 2022
1b983e8
Feat(www): shuttle beta signup (#421)
oddgrd Oct 21, 2022
7c05afc
bug: Fix thruster postgres example (#414)
trezm Oct 21, 2022
9c83baf
feat: shell completions (#343)
XyLyXyRR Oct 21, 2022
c222354
bug: package Secrets.toml (#422)
chesedo Oct 21, 2022
b00671d
bug: big archives being cut off at 32 768 bytes (#423)
chesedo Oct 21, 2022
65b71c8
chore: v0.7.1 (#424)
chesedo Oct 24, 2022
bc13eb7
refactor: provide better context for errors (#430)
chesedo Oct 25, 2022
fb623e7
refactor: do our own health checks on deployer containers (#427)
chesedo Oct 25, 2022
0968b72
bug: clear build folder before extracting (#428)
chesedo Oct 25, 2022
f05fc2b
added some images (#435)
brokad Oct 27, 2022
4be3e5a
feat(www): beta blog updates (#434)
oddgrd Oct 27, 2022
61987e2
feat: add captioned image component (#440)
oddgrd Oct 27, 2022
a5b7634
www: post small tweaks (#439)
brokad Oct 27, 2022
c7a13f9
feat: replace cursed gif (#441)
oddgrd Oct 27, 2022
b430d21
Feat/update contributing (#426)
oddgrd Oct 28, 2022
b097d3b
chore: v0.7.2 (#442)
chesedo Oct 28, 2022
566b9e8
Merge remote-tracking branch 'upstream/main' into development
chesedo Oct 28, 2022
5fd6e40
Merge pull request #443 from shuttle-hq/development
chesedo Oct 28, 2022
c4f0837
feat: link the tracing spans between services (#445)
chesedo Oct 31, 2022
7bfb1a2
Merge remote-tracking branch 'upstream/main' into development
chesedo Oct 31, 2022
c2499cb
Add docker-compose extra flags param in Makefile (#446)
brokad Nov 1, 2022
8a6efb8
misc: restructure repo (#453)
chesedo Nov 2, 2022
6c848bf
refactor: tf files have been moved to shuttle-hq/terraform-aws-shuttle
brokad Nov 2, 2022
bd0c381
fix: wrap around common::ProjectName for parsing (#451)
brokad Nov 3, 2022
9d5e345
fix: gateway state drifts, health checks and project recreation (#447)
brokad Nov 3, 2022
617bad0
misc: add more helpful flags to Makefile
brokad Nov 3, 2022
83cbccd
misc: remove old migrator (#463)
chesedo Nov 4, 2022
b1eee6d
feat: add account_tier column (#458)
brokad Nov 4, 2022
5fbf7c9
feat: prefetch shuttle-service crates (#461)
chesedo Nov 4, 2022
88c877d
Feat: revive via gateway endpoint (#460)
oddgrd Nov 4, 2022
7471c08
feat: create a new admin cli binary crate (#462)
chesedo Nov 4, 2022
b6bd64c
fix: broken link (#467)
oddgrd Nov 8, 2022
b4055af
fix: add timeout to health checks (#468)
brokad Nov 8, 2022
3ab6c71
feat(gateway): add custom domains table and routing (#465)
akrantz01 Nov 8, 2022
3a98a47
WIP feat: count recent start events before restart (#469)
oddgrd Nov 11, 2022
001dbcf
feat: make deployer only answer its own project (#466)
chesedo Nov 11, 2022
836c5f7
Fixed Links in Readme (#477)
Nereuxofficial Nov 13, 2022
3bcc683
refactor: base client error off response status code (#470)
chesedo Nov 14, 2022
e10f096
feat: verify project exists before sending destroy task (#474)
oddgrd Nov 15, 2022
7b80c45
feat: add a custom domains admin route (#473)
chesedo Nov 16, 2022
c3c0ced
bug: deployer freezes (#478)
chesedo Nov 17, 2022
74aeb46
ci: green (#482)
chesedo Nov 17, 2022
3bd6f0f
feat: TLS acceptor with SNI resolver (#471)
brokad Nov 18, 2022
d8fedbd
fix: custom domain routing (#484)
brokad Nov 21, 2022
9a85dc4
refactor: more metrics (#475)
chesedo Nov 21, 2022
ac06f5c
misc: configurable deployment tags (#486)
brokad Nov 21, 2022
b748493
feat: gateway restores removed containers (#485)
brokad Nov 21, 2022
e3fb067
fix: backend bumps and hot fixes (#487)
brokad Nov 23, 2022
57ec829
Feature/support actix web (#491)
biryukovmaxim Nov 30, 2022
80f8e12
Doc: Fix command to prime database with docker-compose (#502)
guerinoni Nov 30, 2022
c2fa52c
Doc: Improve contributing documentation (#499)
guerinoni Nov 30, 2022
6c3025b
feat: static file support for a single folder (#501)
chesedo Dec 1, 2022
6a551d1
ci: build binary (#483)
chesedo Dec 1, 2022
961964a
feat: bump rust to 1.64, bump dependencies (#495)
oddgrd Dec 1, 2022
887dce4
feat: interactive project initialization (#498)
Procrat Dec 2, 2022
ca97f03
feat: bump pinned rust version to 1.65 (#504)
oddgrd Dec 2, 2022
650e3f5
misc: env updates (#509)
chesedo Dec 2, 2022
3d5c55b
feat: make the folder configurable (#508)
chesedo Dec 5, 2022
8052e87
feat: implement workspace inheritance (#506)
oddgrd Dec 7, 2022
d9d6d3e
refactor: switch away from cargo package (#507)
chesedo Dec 7, 2022
b597eef
feat: add spinner wait for `project new` and `project status --follow…
guerinoni Dec 7, 2022
0c4eb94
fix: capitalise correctly (#511)
brokad Dec 7, 2022
6bbda80
fix: make nice (#512)
brokad Dec 7, 2022
2e6ac41
feat: find (soon to be) invalid project names (#479)
chesedo Dec 8, 2022
7525c7a
refactor: missed axum 0.6 update (#513)
chesedo Dec 8, 2022
a37903a
feat: build tests in release profile, limit build workers (#514)
oddgrd Dec 9, 2022
519ac04
feat: create `init` project from correct dir (#518)
oddgrd Dec 9, 2022
a957063
misc: interactive init gif (#519)
brokad Dec 9, 2022
bc7b339
Feat/set examples submodule to main (#520)
oddgrd Dec 9, 2022
5c19ea3
chore: 0.8.0 (#521)
oddgrd Dec 9, 2022
5b9769e
chore: bump examples (#522)
brokad Dec 9, 2022
84250da
bug: hacking static folders (#524)
chesedo Dec 12, 2022
489b925
fix: actix integration with state (#523)
oddgrd Dec 12, 2022
8387138
feat(gateway,deployer): add more tracing events (#500)
akrantz01 Dec 13, 2022
18767f0
feat: canonicalize before trace (#531)
oddgrd Dec 13, 2022
8e93e87
feat: 'clean' subcommand (#530)
chesedo Dec 14, 2022
5c9487d
Feat/set cpu limit (#529)
oddgrd Dec 14, 2022
d60f642
feat: add panamax for mirroring crates.io (#525)
chesedo Dec 14, 2022
5e604b4
feat: build queue (#532)
chesedo Dec 14, 2022
ae8ee01
feat: per-project parallelism (#533)
brokad Dec 14, 2022
a7e7ed6
feat: temp validation of project name in gateway (#534)
oddgrd Dec 14, 2022
24657bc
fix(deployer): keep Cargo.lock between deployments (#517)
Procrat Dec 14, 2022
027b50d
ci: remove build and push req on build binaries (#535)
oddgrd Dec 14, 2022
5ed12ad
refactor: don't crash when failing to release slot (#536)
chesedo Dec 14, 2022
5d638dc
refactor: release build slot parse type correctly (#538)
chesedo Dec 14, 2022
84dd5fa
refactor: remove prefetch (#539)
chesedo Dec 14, 2022
0afa129
feat: add cron job for syncing mirror (#537)
chesedo Dec 14, 2022
998fff7
chore: bump cargo-shuttle to 0.8.1 (#540)
oddgrd Dec 14, 2022
b00b5e8
test: fixes (#554)
chesedo Dec 28, 2022
b1dbdb7
Refactor: remove deprecated auth command (#550)
oddgrd Dec 29, 2022
2d0f338
ci: add Makefile command for Windows to convert .sh files to LF forma…
joshua-mo-143 Dec 30, 2022
45eadce
Fix: dependencies compiled with incompatible versions of rustc (#545)
oddgrd Dec 30, 2022
cb342fd
feat: create subcommand to list all projects of calling account (#553)
emmakuen Dec 30, 2022
eda4769
bug: deployer drifting state (#548)
chesedo Jan 3, 2023
d113ca1
ci: fix bin builds (#546)
oddgrd Jan 5, 2023
85268c9
update contributing (#556)
olekspickle Jan 6, 2023
0599a13
feat: Support Poise (#560)
Anafabula Jan 9, 2023
8f71804
feat: add flag for router IP local run (#565)
joshua-mo-143 Jan 10, 2023
604271a
bug: no networks (#541)
chesedo Jan 10, 2023
8414714
Merge remote-tracking branch 'upstream/main' into shuttle-next
chesedo Jan 16, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
refactor: more metrics (#475)
* feat: add more metric dimensions to deployer

* feat: add more metric dimensions to gateway

* refactor: common metrics code

* refactor: forward account name

* refactor: add backend feature to deployer

* refactor: standardize naming

* refactor: cargo sort
chesedo authored Nov 21, 2022
commit 9a85dc44daa3529f8d51c88ac44b118183301e4b
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions common/Cargo.toml
Original file line number Diff line number Diff line change
@@ -9,6 +9,7 @@ description = "Common library for the shuttle platform (https://www.shuttle.rs/)
[dependencies]
anyhow = { version = "1.0.62", optional = true }
async-trait = { version = "0.1.52", optional = true }
axum = { version = "0.5.8", optional = true }
chrono = { version = "0.4.22", features = ["serde"] }
comfy-table = { version = "6.1.0", optional = true }
crossterm = { version = "0.25.0", optional = true }
@@ -25,5 +26,6 @@ uuid = { version = "1.1.1", features = ["v4", "serde"] }
[features]
default = ["models"]

backend = ["async-trait", "axum"]
display = ["comfy-table", "crossterm"]
models = ["anyhow", "async-trait", "display", "http", "reqwest", "serde_json"]
33 changes: 33 additions & 0 deletions common/src/backends/metrics.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
use std::{collections::HashMap, convert::Infallible};

use async_trait::async_trait;
use axum::extract::{FromRequest, Path, RequestParts};
use tracing::Span;

/// Used to record a bunch of metrics info
/// The tracing layer on the server should record a `request.params.<param>` field for each parameter
/// that should be recorded
pub struct Metrics;

#[async_trait]
impl<B> FromRequest<B> for Metrics
where
B: Send,
{
type Rejection = Infallible;

async fn from_request(req: &mut RequestParts<B>) -> Result<Self, Self::Rejection> {
// Get path parameters if they exist
let Path(path): Path<HashMap<String, String>> = match req.extract().await {
Ok(path) => path,
Err(_) => return Ok(Metrics),
};

let span = Span::current();

for (param, value) in path {
span.record(format!("request.params.{param}").as_str(), value);
}
Ok(Metrics)
}
}
1 change: 1 addition & 0 deletions common/src/backends/mod.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
pub mod metrics;
2 changes: 2 additions & 0 deletions common/src/lib.rs
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
#[cfg(feature = "backend")]
pub mod backends;
pub mod database;
pub mod deployment;
pub mod log;
1 change: 1 addition & 0 deletions deployer/Cargo.toml
Original file line number Diff line number Diff line change
@@ -45,6 +45,7 @@ uuid = { version = "1.1.2", features = ["v4"] }
[dependencies.shuttle-common]
version = "0.7.2"
path = "../common"
features = ["backend"]

[dependencies.shuttle-proto]
version = "0.7.2"
44 changes: 39 additions & 5 deletions deployer/src/handlers/mod.rs
Original file line number Diff line number Diff line change
@@ -2,7 +2,7 @@ mod error;

use axum::body::{Body, BoxBody};
use axum::extract::ws::{self, WebSocket};
use axum::extract::{Extension, Path, Query};
use axum::extract::{Extension, MatchedPath, Path, Query};
use axum::http::{Request, Response};
use axum::middleware::from_extractor;
use axum::routing::{get, Router};
@@ -13,6 +13,7 @@ use fqdn::FQDN;
use futures::StreamExt;
use opentelemetry::global;
use opentelemetry_http::HeaderExtractor;
use shuttle_common::backends::metrics::Metrics;
use shuttle_common::models::secret;
use shuttle_common::project::ProjectName;
use shuttle_common::LogItem;
@@ -45,7 +46,10 @@ pub fn make_router(
"/projects/:project_name/services/:service_name",
get(get_service).post(post_service).delete(delete_service),
)
.route("/projects/:project_name/services/:service_name/summary", get(get_service_summary))
.route(
"/projects/:project_name/services/:service_name/summary",
get(get_service_summary),
)
.route(
"/projects/:project_name/deployments/:deployment_id",
get(get_deployment).delete(delete_deployment),
@@ -54,7 +58,10 @@ pub fn make_router(
"/projects/:project_name/ws/deployments/:deployment_id/logs",
get(get_logs_subscribe),
)
.route("/projects/:project_name/deployments/:deployment_id/logs", get(get_logs))
.route(
"/projects/:project_name/deployments/:deployment_id/logs",
get(get_logs),
)
.route(
"/projects/:project_name/secrets/:service_name",
get(get_secrets),
@@ -65,10 +72,34 @@ pub fn make_router(
.layer(RequireAuthorizationLayer::bearer(&admin_secret))
// This route should be below the auth bearer since it does not need authentication
.route("/projects/:project_name/status", get(get_status))
.route_layer(from_extractor::<Metrics>())
.layer(
TraceLayer::new_for_http()
.make_span_with(|request: &Request<Body>| {
let span = debug_span!("request", http.uri = %request.uri(), http.method = %request.method(), http.status_code = field::Empty);
let path = if let Some(path) = request.extensions().get::<MatchedPath>() {
path.as_str()
} else {
""
};

let account_name = request
.headers()
.get("X-Shuttle-Account-Name")
.map(|value| value.to_str().unwrap_or_default());

let span = debug_span!(
"request",
http.uri = %request.uri(),
http.method = %request.method(),
http.status_code = field::Empty,
account.name = account_name,
// A bunch of extra things for metrics
// Should be able to make this clearer once `Valuable` support lands in tracing
request.path = path,
request.params.project_name = field::Empty,
request.params.service_name = field::Empty,
request.params.deployment_id = field::Empty,
);
let parent_context = global::get_text_map_propagator(|propagator| {
propagator.extract(&HeaderExtractor(request.headers()))
});
@@ -79,7 +110,10 @@ pub fn make_router(
.on_response(
|response: &Response<BoxBody>, latency: Duration, span: &Span| {
span.record("http.status_code", &response.status().as_u16());
debug!(latency = format_args!("{} ns", latency.as_nanos()), "finished processing request");
debug!(
latency = format_args!("{} ns", latency.as_nanos()),
"finished processing request"
);
},
),
)
6 changes: 1 addition & 5 deletions gateway/Cargo.toml
Original file line number Diff line number Diff line change
@@ -7,24 +7,19 @@ publish = false
[dependencies]
acme2 = "0.5.1"
async-trait = "0.1.52"

axum = { version = "0.5.8", features = [ "headers" ] }
axum-server = { version = "0.4.4", features = [ "tls-rustls" ] }

base64 = "0.13"
bollard = "0.13"
chrono = "0.4"
clap = { version = "4.0.0", features = [ "derive" ] }

fqdn = "0.2.2"

futures = "0.3.21"
http = "0.2.8"
hyper = { version = "0.14.19", features = [ "stream" ] }
# not great, but waiting for WebSocket changes to be merged
hyper-reverse-proxy = { git = "https://github.com/chesedo/hyper-reverse-proxy", branch = "bug/host_header" }
instant-acme = "0.1.0"

lazy_static = "1.4.0"
once_cell = "1.14.0"
opentelemetry = { version = "0.18.0", features = ["rt-tokio"] }
@@ -50,6 +45,7 @@ uuid = { version = "1.2.1", features = [ "v4" ] }
[dependencies.shuttle-common]
version = "0.7.2"
path = "../common"
features = ["backend"]

[dev-dependencies]
anyhow = "1"
38 changes: 30 additions & 8 deletions gateway/src/api/latest.rs
Original file line number Diff line number Diff line change
@@ -4,8 +4,9 @@ use std::sync::Arc;
use std::time::Duration;

use axum::body::{Body, BoxBody};
use axum::extract::{Extension, Path};
use axum::extract::{Extension, MatchedPath, Path};
use axum::http::Request;
use axum::middleware::from_extractor;
use axum::response::Response;
use axum::routing::{any, get, post};
use axum::{Json as AxumJson, Router};
@@ -14,6 +15,7 @@ use futures::Future;
use http::StatusCode;
use instant_acme::{AccountCredentials, ChallengeType};
use serde::{Deserialize, Serialize};
use shuttle_common::backends::metrics::Metrics;
use shuttle_common::models::error::ErrorKind;
use shuttle_common::models::{project, user};
use tokio::sync::mpsc::Sender;
@@ -151,10 +153,10 @@ async fn delete_project(

async fn route_project(
Extension(service): Extension<Arc<GatewayService>>,
ScopedUser { scope, .. }: ScopedUser,
scoped_user: ScopedUser,
req: Request<Body>,
) -> Result<Response<Body>, Error> {
service.route(&scope, req).await
service.route(&scoped_user, req).await
}

async fn get_status(Extension(sender): Extension<Sender<BoxedTask>>) -> Response<Body> {
@@ -310,15 +312,35 @@ impl ApiBuilder {
}

pub fn with_default_traces(mut self) -> Self {
self.router = self.router.layer(
self.router = self.router.route_layer(from_extractor::<Metrics>()).layer(
TraceLayer::new_for_http()
.make_span_with(|request: &Request<Body>| {
debug_span!("request", http.uri = %request.uri(), http.method = %request.method(), http.status_code = field::Empty, account.name = field::Empty, account.project = field::Empty)
let path = if let Some(path) = request.extensions().get::<MatchedPath>() {
path.as_str()
} else {
""
};

debug_span!(
"request",
http.uri = %request.uri(),
http.method = %request.method(),
http.status_code = field::Empty,
account.name = field::Empty,
// A bunch of extra things for metrics
// Should be able to make this clearer once `Valuable` support lands in tracing
request.path = path,
request.params.project_name = field::Empty,
request.params.account_name = field::Empty,
)
})
.on_response(
|response: &Response<BoxBody>, latency: Duration, span: &Span| {
span.record("http.status_code", response.status().as_u16());
debug!(latency = format_args!("{} ns", latency.as_nanos()), "finished processing request");
debug!(
latency = format_args!("{} ns", latency.as_nanos()),
"finished processing request"
);
},
),
);
@@ -330,11 +352,11 @@ impl ApiBuilder {
.router
.route("/", get(get_status))
.route(
"/projects/:project",
"/projects/:project_name",
get(get_project).delete(delete_project).post(post_project),
)
.route("/users/:account_name", get(get_user).post(post_user))
.route("/projects/:project/*any", any(route_project))
.route("/projects/:project_name/*any", any(route_project))
.route("/admin/revive", post(revive_projects));
self
}
2 changes: 0 additions & 2 deletions gateway/src/auth.rs
Original file line number Diff line number Diff line change
@@ -247,8 +247,6 @@ where
.unwrap(),
};

// Record current project for tracing purposes
Span::current().record("account.project", &scope.to_string());
if user.is_super_user() || user.projects.contains(&scope) {
Ok(Self { user, scope })
} else {
14 changes: 11 additions & 3 deletions gateway/src/service.rs
Original file line number Diff line number Diff line change
@@ -8,6 +8,7 @@ use axum::response::Response;
use bollard::network::ListNetworksOptions;
use bollard::{Docker, API_DEFAULT_VERSION};
use fqdn::Fqdn;
use http::HeaderValue;
use hyper::client::connect::dns::GaiResolver;
use hyper::client::HttpConnector;
use hyper::Client;
@@ -25,7 +26,7 @@ use tracing_opentelemetry::OpenTelemetrySpanExt;

use crate::acme::CustomDomain;
use crate::args::ContextArgs;
use crate::auth::{Key, Permissions, User};
use crate::auth::{Key, Permissions, ScopedUser, User};
use crate::project::Project;
use crate::task::TaskBuilder;
use crate::{AccountName, DockerContext, Error, ErrorKind, ProjectName};
@@ -205,9 +206,10 @@ impl GatewayService {

pub async fn route(
&self,
project_name: &ProjectName,
scoped_user: &ScopedUser,
mut req: Request<Body>,
) -> Result<Response<Body>, Error> {
let project_name = &scoped_user.scope;
let target_ip = self
.find_project(project_name)
.await?
@@ -223,9 +225,15 @@ impl GatewayService {

debug!(target_url, "routing control");

let headers = req.headers_mut();
headers.append(
"X-Shuttle-Account-Name",
HeaderValue::from_str(&scoped_user.user.name.to_string()).unwrap(),
);

let cx = Span::current().context();
global::get_text_map_propagator(|propagator| {
propagator.inject_context(&cx, &mut HeaderInjector(req.headers_mut()))
propagator.inject_context(&cx, &mut HeaderInjector(headers))
});

let resp = PROXY_CLIENT