Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature/SK-1265 | Implement SQL store #785

Open
wants to merge 65 commits into
base: master
Choose a base branch
from
Open

Feature/SK-1265 | Implement SQL store #785

wants to merge 65 commits into from

Conversation

niklastheman
Copy link
Contributor

Implement SQL store

In order to implement sql stores for all entities some refactoring was needed:

  • MongoStateStore has been removed. All db queries are now done via the separate stores.
  • The API interface class has been removed. It was dependent upon mongo state store.
  • All the legacy endpoints in root (with the exception of /health and /get_controller_status) has been deprecated and will return 410 if called.
  • All previously missing required endpoints such ass add_client has been moved to api/v1
  • Client V1 has been removed. It used a lot of the legacy endpoints.
  • Client V2 (current) only uses /api/v1 endpoints
  • Legacy start session has been removed.
  • SQLAlchemy and psycopg2 dependencies added.
  • Postgres service added in the docker compose file.
  • A lot of API-tests has been removed as they are no longer relevant. Plans are in place to write new once for the /api/v1 routes.

This implementation enables running FEDn with an SQL-database. It makes use of an ORM called SQLAlchemy. This way FEDn can run with a number of different SQL-databases including PostgreSQL and SQLite.

niklastheman and others added 30 commits December 20, 2024 15:01
@niklastheman
added graphql server
2710e6e
@niklastheman
sessions added
bab877e
@niklastheman
get single session
9c4ad30
@niklastheman
added models
0fe3cfd
@niklastheman
added validations
a29f0a1
@niklastheman
statuses added
750cf3a
@niklastheman
added plura/singular naming to follow convention
6dcb6af
@niklastheman
added abstract parent class, Store along with MongoDBStore
0e596dc
@niklastheman
removed use_typings option
9271df6
@niklastheman
clean up
e55ad72
@niklastheman
merged SK-1109
b069844
@niklastheman
removed use_typings, changes to complie with feature/SK-1109
0ec6d6c
@niklastheman
started on replacing mongostatestore
4ebb1eb
@niklastheman
removed get_combiners from mongostatestore and started using combiner…
eaf75c0 
…_store in Network class
@niklastheman
removed add/delete combiner from network and mongostatestore
08f8583
@niklastheman
moved get and list clients from mongostatestore
fab412a
@niklastheman
update sessions with session_store
328217b
@niklastheman
removed set_session_status from mongostatestore
5a5303f
@niklastheman
added round_store to combiner
1ac6880
@niklastheman
clean up
9d9bfa8
@niklastheman
upload model moved to api/v1/models
010528f
@niklastheman
removed get model
d0df714
@niklastheman
moved add client to api/v1/clients/add
70fc83d
@niklastheman
moved download package to api/v1/packages/download
d542140
@niklastheman
moved get checksum to api/v1/packages/checksum
a1ac5b1
@niklastheman
removed old client and old package runtime class
839bf39
@niklastheman
mongostatestore dependecy to control removed and repository dependecy…
db14426 
… injected into control
@niklastheman
get_client_config moved to api/v1/clients/config
57ba8d1
@niklastheman
removed API (interface) class
5fdc75c
@niklastheman
added indexes to mogno stores
64fdd83
@niklastheman
started on models sql store
62b8a4c
@niklastheman
sql model store works for training
a47616a
@niklastheman
added ancestors and descendants to sql models store and thus model st…
14fb7d0 
…ore v1
@niklastheman
added postgres
9ea7304
@niklastheman
started on rounds sql store
2df0655
@niklastheman
added sql round store
caa1add
@niklastheman
added get_latest_round_id to round store
6a6b283
@niklastheman
added session and model id as foregin keys in round config model
420e12f
@niklastheman
added status store
420f347
@niklastheman
added validations sql store
099ff2e
@niklastheman
added client store
dbf3f5a
@niklastheman
started on prediction store
fd1c34b
@niklastheman
prediction store v1
633d7cf
@niklastheman
create sql tables only if sql stores
51c463c
@niklastheman
linting fix
6920404
@niklastheman
sqllite postgres or mongo based on env:s. moved table creation to cot…
34a5818 
…roller only
@niklastheman
implemented connected_client_count for client sql store
a7d7e08
@niklastheman
clean up
361d967
@niklastheman
client_store update should be like every other update
b8f89b7
@niklastheman
run table creation script from combiner to fix timing issue
669c154
github-advanced-security[bot]
github-advanced-security bot found potential problems Jan 9, 2025
View reviewed changes
fedn/network/api/v1/client_routes.py Fixed Show fixed Hide fixed
fedn/network/api/v1/model_routes.py Fixed Show fixed Hide fixed
fedn/network/api/v1/package_routes.py Fixed Show fixed Hide fixed
fedn/network/storage/statestore/stores/client_store.py Fixed Show fixed Hide fixed
fedn/network/storage/statestore/stores/combiner_store.py Fixed Show fixed Hide fixed
fedn/network/storage/statestore/stores/prediction_store.py Fixed Show fixed Hide fixed
fedn/network/storage/statestore/stores/round_store.py Fixed Show fixed Hide fixed
fedn/network/storage/statestore/stores/session_store.py Fixed Show fixed Hide fixed
fedn/network/storage/statestore/stores/status_store.py Fixed Show fixed Hide fixed
fedn/network/storage/statestore/stores/validation_store.py Fixed Show fixed Hide fixed
@Wrede Wrede added HOLD feature New feature or request labels Jan 10, 2025
@github-actions github-actions bot added the minor label Jan 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ahellander ahellander Awaiting requested review from ahellander

@Wrede Wrede Awaiting requested review from Wrede

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
feature New feature or request HOLD minor
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@niklastheman @Wrede