Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Feature/poem migration start | NPG-0000 #538

Closed
wants to merge 42 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
863aec8
build(event-db): remove obsolete .env file causing warnings.
stevenj Aug 23, 2023
6f73f0f
build(cat-data-service): Update dependencies and add dependencies on …
stevenj Aug 23, 2023
c58c88f
feat(cat-data-service): Run a Poem service in parallel with the curre…
stevenj Aug 23, 2023
8d8005f
feat(cat-data-service): Add stoplight as the default docs viewer for …
stevenj Aug 23, 2023
9643163
feat(cat-data-service): Add prometheus metrics to poem service
stevenj Aug 25, 2023
416dbc1
Merge branch 'main' into feature/poem-migration-start
stevenj Aug 26, 2023
5da3cc9
feat(local environment): Add basic monitoring stack with working log …
stevenj Aug 28, 2023
ebb5644
feat(cat-data-service): refactor and cleanup the poem api integration
stevenj Aug 28, 2023
a411a78
chore(local environment): Properly exclude `local` from only the repo…
stevenj Aug 28, 2023
96a8a10
fix(cicd): Fix opentelemetry version to 0.19 to prevent version issues
stevenj Aug 28, 2023
7e7cb61
fix fmt
Mr-Leshiy Aug 28, 2023
e3197aa
refactor, fix issues
Mr-Leshiy Aug 28, 2023
0c24f52
feat(cat-data-service): wip code cleanups
stevenj Aug 28, 2023
0b6a018
Merge branch 'feature/poem-migration-start' of github.com:input-outpu…
stevenj Aug 28, 2023
493be6a
feat(cat-data-service): Further cleanup for poem integration
stevenj Aug 29, 2023
edf8b38
feat(cat-data-service): Further cleanup for poem integration
stevenj Aug 29, 2023
ba21941
Merge branch 'main' into feature/poem-migration-start
stevenj Aug 29, 2023
643ac1b
feat(cat-data-service): More wip initial poem integration work
stevenj Aug 30, 2023
5dd41f5
docs(cat-data-service): Add documentation for all currently used resp…
stevenj Aug 30, 2023
c717d8e
feat(snapshot): Add analysis to snapshot check to look for and catego…
stevenj Sep 1, 2023
ccfe052
feat(cat-data-service): Add API env vars to set the hostnames
stevenj Sep 1, 2023
799912a
feat(cat-data-service): Move the panic handling to its own module
stevenj Sep 1, 2023
5a9c9bf
feat(cat-data-service): Add full documentation for the API top level
stevenj Sep 1, 2023
6ee6a66
feat(cat-data-service): Add health endpoints and document them.
stevenj Sep 1, 2023
9f82ef1
feat(cat-data-service): Add generic parts of the api
stevenj Sep 1, 2023
e2b619c
feat(cat-data-service): try and make spans work and async aware
stevenj Sep 1, 2023
e1b0222
Merge branch 'main' into feature/poem-migration-start
stevenj Sep 3, 2023
5f289c5
fix(cat-data-service): fix getting service url from env var
stevenj Sep 3, 2023
99cd8d7
style(cat-data-service): cleanup clippy lints
stevenj Sep 3, 2023
bd2406b
docs(cat-data-service): Fix quoting in documentation causing doc buil…
stevenj Sep 3, 2023
3b60030
feat(cat-data-service): Refactor ready to call axum directly when poe…
stevenj Sep 4, 2023
8c0e9be
feat(cat-data-service): Internally proxy all unimplemented endpoints …
stevenj Sep 4, 2023
65dee5f
feat(cat-data-service): Fix prometheus on upgrade and use middleware …
stevenj Sep 4, 2023
a0b7e42
refactor(cat-data-service): Cleanup API Returns so that they only nee…
stevenj Sep 5, 2023
c038f06
feat(cat-data-service): Logging, Metrics and tracing now works correc…
stevenj Sep 7, 2023
3f5f4c2
Merge branch 'main' into feature/poem-migration-start
stevenj Sep 8, 2023
3e8c22a
fix(cat-data-service): Fix logging so its enabled outside spans. Als…
stevenj Sep 8, 2023
2e272fc
feat(cat-data-service): New endpoint tests
stevenj Sep 8, 2023
a5b2305
Merge branch 'feature/poem-migration-start' of github.com:input-outpu…
stevenj Sep 8, 2023
b6ea6c9
feat(cat-data-service): Finish validation example for demo.
stevenj Sep 8, 2023
77185a1
feat: Add `registration/voter` endpoint to poem service | NPG-8137 (#…
Mr-Leshiy Sep 19, 2023
a46f19c
feat: Handle DB connection timeouts | NPG-000 (#590)
saibatizoku Oct 17, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
feat(cat-data-service): Internally proxy all unimplemented endpoints …
…to the axum processing.
stevenj committed Sep 4, 2023
commit 8c0e9be086ca43cc5c56c454bde931e5e4729033
2 changes: 2 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 3 additions & 1 deletion src/cat-data-service/Cargo.toml
Original file line number Diff line number Diff line change
@@ -52,9 +52,11 @@ url = { workspace = true }
dotenvy = { workspace = true }
panic-message = "0.3"
regex = { workspace = true }
tower = { version = "0.4", features = ["util"] }
hyper = "0.14"
bytes = "1.4"

[dev-dependencies]
tower = { version = "0.4", features = ["util"] }
quickcheck = { version = "0.9" }
quickcheck_macros = { version = "0.9" }

Original file line number Diff line number Diff line change
@@ -44,7 +44,7 @@ async fn live_exec() -> Result<bool, Error> {
/// https://github.com/input-output-hk/catalyst-core/tree/main/src/event-db/Readme.md
#[cfg(test)]
mod tests {
use crate::{axum::app, state::State};
use crate::{axum_service::app, state::State};
use axum::{
body::{Body, HttpBody},
http::{Request, StatusCode},
File renamed without changes.
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
use crate::{axum::handle_result, service::Error, state::State, types::SerdeType};
use crate::{axum_service::handle_result, service::Error, state::State, types::SerdeType};
use axum::{routing::get, Router};
use event_db::types::vit_ss::fund::FundWithNext;
use std::sync::Arc;
@@ -35,7 +35,7 @@ async fn fund_exec(state: Arc<State>) -> Result<SerdeType<FundWithNext>, Error>
#[cfg(test)]
mod tests {
use super::*;
use crate::service::{app, tests::response_body_to_json};
use crate::axum_service::{app, tests::response_body_to_json};
use axum::{
body::Body,
http::{Request, StatusCode},
File renamed without changes.
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
use crate::{axum::handle_result, service::Error, state::State, types::SerdeType};
use crate::{axum_service::handle_result, service::Error, state::State, types::SerdeType};
use axum::{extract::Path, routing::get, Router};
use event_db::types::{ballot::ObjectiveBallots, event::EventId};
use std::sync::Arc;
@@ -44,7 +44,7 @@ async fn ballots_exec(
#[cfg(test)]
mod tests {
use super::*;
use crate::service::{app, tests::response_body_to_json};
use crate::axum_service::{app, tests::response_body_to_json};
use axum::{
body::Body,
http::{Request, StatusCode},
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
use super::LimitOffset;
use crate::{axum::handle_result, service::Error, state::State, types::SerdeType};
use crate::{axum_service::handle_result, service::Error, state::State, types::SerdeType};
use axum::{
extract::{Path, Query},
routing::get,
@@ -83,7 +83,7 @@ async fn events_exec(
#[cfg(test)]
mod tests {
use super::*;
use crate::service::{app, tests::response_body_to_json};
use crate::axum_service::{app, tests::response_body_to_json};
use axum::{
body::Body,
http::{Request, StatusCode},
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
use crate::{axum::handle_result, service::Error, state::State, types::SerdeType};
use crate::{axum_service::handle_result, service::Error, state::State, types::SerdeType};
use axum::{extract::Path, routing::get, Router};
use event_db::types::{ballot::ProposalBallot, event::EventId, objective::ObjectiveId};
use std::sync::Arc;
@@ -51,7 +51,7 @@ async fn ballots_exec(
#[cfg(test)]
mod tests {
use super::*;
use crate::service::{app, tests::response_body_to_json};
use crate::axum_service::{app, tests::response_body_to_json};
use axum::{
body::Body,
http::{Request, StatusCode},
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
use crate::{
axum::{handle_result, v1::LimitOffset},
axum_service::{handle_result, v1::LimitOffset},
service::Error,
state::State,
types::SerdeType,
@@ -144,7 +144,7 @@ async fn objectives_voting_statuses_exec(
#[cfg(test)]
mod tests {
use super::*;
use crate::service::{app, tests::response_body_to_json};
use crate::axum_service::{app, tests::response_body_to_json};
use axum::{
body::Body,
http::{Request, StatusCode},
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
use crate::{axum::handle_result, service::Error, state::State, types::SerdeType};
use crate::{axum_service::handle_result, service::Error, state::State, types::SerdeType};
use axum::{extract::Path, routing::get, Router};
use event_db::types::{
ballot::Ballot, event::EventId, objective::ObjectiveId, proposal::ProposalId,
@@ -53,7 +53,7 @@ async fn ballot_exec(
#[cfg(test)]
mod tests {
use super::*;
use crate::service::{app, tests::response_body_to_json};
use crate::axum_service::{app, tests::response_body_to_json};
use axum::{
body::Body,
http::{Request, StatusCode},
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
use crate::{
axum::{handle_result, v1::LimitOffset},
axum_service::{handle_result, v1::LimitOffset},
service::Error,
state::State,
types::SerdeType,
@@ -110,7 +110,7 @@ async fn proposal_exec(
#[cfg(test)]
mod tests {
use super::*;
use crate::service::{app, tests::response_body_to_json};
use crate::axum_service::{app, tests::response_body_to_json};
use axum::{
body::Body,
http::{Request, StatusCode},
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
use crate::{
axum::{handle_result, v1::LimitOffset},
axum_service::{handle_result, v1::LimitOffset},
service::Error,
state::State,
types::SerdeType,
@@ -65,7 +65,7 @@ async fn reviews_exec(
#[cfg(test)]
mod tests {
use super::*;
use crate::service::{app, tests::response_body_to_json};
use crate::axum_service::{app, tests::response_body_to_json};
use axum::{
body::Body,
http::{Request, StatusCode},
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
use crate::{
axum::{handle_result, v1::LimitOffset},
axum_service::{handle_result, v1::LimitOffset},
service::Error,
state::State,
types::SerdeType,
@@ -63,7 +63,7 @@ async fn review_types_exec(
#[cfg(test)]
mod tests {
use super::*;
use crate::service::{app, tests::response_body_to_json};
use crate::axum_service::{app, tests::response_body_to_json};
use axum::{
body::Body,
http::{Request, StatusCode},
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
use crate::{
axum::handle_result,
axum_service::handle_result,
service::Error,
state::State,
types::jorm_mock::{AccountId, AccountVote, Fragments, FragmentsProcessingSummary},
@@ -62,7 +62,7 @@ async fn account_votes_exec(
#[cfg(test)]
mod tests {
use super::*;
use crate::service::{app, tests::response_body_to_json};
use crate::axum_service::{app, tests::response_body_to_json};
use axum::{
body::Body,
http::{header, Method, Request, StatusCode},
File renamed without changes.
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
use crate::{axum::handle_result, service::Error, state::State, types::SerdeType};
use crate::{axum_service::handle_result, service::Error, state::State, types::SerdeType};
use axum::{
extract::{Path, Query},
routing::get,
@@ -104,7 +104,7 @@ async fn delegations_exec(
#[cfg(test)]
mod tests {
use super::*;
use crate::service::{app, tests::response_body_to_json};
use crate::axum_service::{app, tests::response_body_to_json};
use axum::{
body::Body,
http::{Request, StatusCode},
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
use crate::{axum::handle_result, service::Error, state::State, types::SerdeType};
use crate::{axum_service::handle_result, service::Error, state::State, types::SerdeType};
use axum::{extract::Query, routing::post, Json, Router};
use event_db::types::search::{SearchQuery, SearchResult};
use serde::Deserialize;
@@ -58,7 +58,7 @@ async fn search_exec(
#[cfg(test)]
mod tests {
use super::*;
use crate::service::{app, tests::response_body_to_json};
use crate::axum_service::{app, tests::response_body_to_json};
use axum::{
body::Body,
http::{header, Method, Request, StatusCode},
2 changes: 1 addition & 1 deletion src/cat-data-service/src/main.rs
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
use clap::Parser;

mod axum;
mod axum_service;
mod cli;
mod logger;
mod service;
9 changes: 5 additions & 4 deletions src/cat-data-service/src/service/mod.rs
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
//! Main entrypoint to the service
//!
use crate::axum;
use crate::axum_service;
use crate::state::State;

use serde::Serialize;
@@ -14,7 +14,7 @@ mod docs;

// These modules are utility or generic types/functions
mod generic;
mod service;
mod poem_service;
mod utilities;

#[derive(thiserror::Error, Debug)]
@@ -64,9 +64,10 @@ pub async fn run(
let mut axum_service = *service_addr;
axum_service.set_port(axum_service.port() + 1);

// This can be simplified to an .await when axum is finally removed.
try_join!(
axum::run(&axum_service, metrics_addr, state),
service::run(service_addr),
axum_service::run(&axum_service, metrics_addr, state.clone()),
poem_service::run(service_addr, state),
)?;

Ok(())
Original file line number Diff line number Diff line change
@@ -11,15 +11,21 @@ use crate::service::utilities::metrics_tracing::{init_prometheus, log_requests};
use crate::service::utilities::middleware::chain_axum::ChainAxum;

use crate::settings::{get_api_hostnames, API_URL_PREFIX};
use crate::state::State;

use poem::endpoint::PrometheusExporter;
use poem::listener::TcpListener;
use poem::middleware::{CatchPanic, Cors, OpenTelemetryMetrics};
use poem::{EndpointExt, IntoEndpoint, Route};
use std::net::SocketAddr;
use std::sync::Arc;

/// This exists to allow us to add extra routes to the service for testing purposes.
pub(crate) fn mk_app(hosts: Vec<String>, base_route: Option<Route>) -> impl IntoEndpoint {
pub(crate) fn mk_app(
hosts: Vec<String>,
base_route: Option<Route>,
state: Arc<State>,
) -> impl IntoEndpoint {
// Get the base route if defined, or a new route if not.
let base_route = match base_route {
Some(route) => route,
@@ -42,6 +48,7 @@ pub(crate) fn mk_app(hosts: Vec<String>, base_route: Option<Route>) -> impl Into
.with(OpenTelemetryMetrics::new())
.with(ChainAxum::new()) // TODO: Remove this once all endpoints are ported.
.with(CatchPanic::new().with_handler(ServicePanicHandler))
.data(state)
.around(|ep, req| async move { Ok(log_requests(ep, req).await) })
}

@@ -59,7 +66,7 @@ pub(crate) fn mk_app(hosts: Vec<String>, base_route: Option<Route>) -> impl Into
/// * `Error::EventDbError` - cannot connect to the event db
/// * `Error::IoError` - An IO error has occurred.
///
pub async fn run(addr: &SocketAddr) -> Result<(), Error> {
pub async fn run(addr: &SocketAddr, state: Arc<State>) -> Result<(), Error> {
tracing::info!("Starting Poem Service ...");
tracing::info!("Listening on {addr}");

@@ -71,7 +78,7 @@ pub async fn run(addr: &SocketAddr) -> Result<(), Error> {

let hosts = get_api_hostnames(addr);

let app = mk_app(hosts, None);
let app = mk_app(hosts, None, state);

poem::Server::new(TcpListener::bind(addr))
.run(app)
Loading