Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: badges with category and badge_type fields #201

Merged
merged 55 commits into from
Sep 17, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
60e9c92
setting things up and experimenting with creating new endpoint
allisonsuarez Aug 21, 2020
d1a83c0
fixed issue with other proxys
allisonsuarez Aug 21, 2020
8de7b3f
wrote queries, the code is so similar to tags that I see why it might…
allisonsuarez Aug 22, 2020
9f1bfd5
working BadgeAPI..?
allisonsuarez Aug 25, 2020
1c803db
PUT not working, get seems fine so far
allisonsuarez Aug 26, 2020
5639e44
adding more to API
allisonsuarez Aug 26, 2020
fb2a5ae
able to call resource/badge/badge_name now, having errors but the rou…
allisonsuarez Aug 27, 2020
4580db2
merge conf resolved
allisonsuarez Aug 28, 2020
2c11840
small fix on init
allisonsuarez Aug 27, 2020
a0bfcee
merge conf resolved
allisonsuarez Aug 27, 2020
70502fd
ITS ALIVEgit status! PUT badges in resource works (at least for table…
allisonsuarez Aug 28, 2020
0062ee1
added logger to add badge, badges not showing up on resource detail GET
allisonsuarez Aug 28, 2020
7e40635
GET badges works
allisonsuarez Aug 28, 2020
db3e8c2
get table now returns badges properly on tables :D
allisonsuarez Aug 28, 2020
6358dab
Got badges on dashboards
allisonsuarez Aug 28, 2020
36415b5
DELETE says it happened successful but badge still showing up on GET …
allisonsuarez Sep 1, 2020
1ca62a4
small error in neo4j proxy query, fixed, delete works now
allisonsuarez Sep 1, 2020
e15ae5e
lint fixes
allisonsuarez Sep 1, 2020
ec82a24
added unit tests for badge api, will update other tests once new func…
allisonsuarez Sep 1, 2020
f025539
added more unit tests
allisonsuarez Sep 1, 2020
324b7db
removed new badges, just badges now
allisonsuarez Sep 1, 2020
5f48427
updated TagAPI to not accept badges
allisonsuarez Sep 1, 2020
8fdb815
fixed all unit tests
allisonsuarez Sep 2, 2020
25a12aa
still trying to figure out how to implement exclusivity of categories
allisonsuarez Sep 2, 2020
c2f759d
implemented combination whitelist check
allisonsuarez Sep 3, 2020
1e2debf
added config with acceptable badges and init for modules
allisonsuarez Sep 3, 2020
13dc812
changed things around a bit
allisonsuarez Sep 3, 2020
2b31538
updated mock import
allisonsuarez Sep 3, 2020
2dbf0a6
small changes, updated import
allisonsuarez Sep 8, 2020
d7a43e4
merge conflict
allisonsuarez Aug 21, 2020
02c8576
merge conflict
allisonsuarez Aug 21, 2020
74423bd
merge conflict
feng-tao Aug 21, 2020
499e788
merge conf
allisonsuarez Aug 21, 2020
12e666d
fixed tests to use new config
allisonsuarez Sep 8, 2020
884c729
merge conf
allisonsuarez Sep 8, 2020
63f06cf
atlas
allisonsuarez Sep 16, 2020
60d1f40
atlas changes
allisonsuarez Sep 16, 2020
32e111f
lint
allisonsuarez Sep 16, 2020
b5e8d83
config
allisonsuarez Sep 8, 2020
1159be1
requirements common version bump
allisonsuarez Sep 16, 2020
65c0d79
removed print
allisonsuarez Sep 16, 2020
550169c
reverted get tags change to mater state
allisonsuarez Sep 16, 2020
c91b0cf
logic changes
allisonsuarez Sep 11, 2020
11ad3c2
fixed tests for table badges
allisonsuarez Sep 10, 2020
761fb2f
fixing tests
allisonsuarez Sep 16, 2020
e13a7a8
messages updated
allisonsuarez Sep 10, 2020
ff21ceb
Badge type fixes
allisonsuarez Sep 10, 2020
6e737d5
fixed tests for table badges
allisonsuarez Sep 10, 2020
326fba1
records to record
allisonsuarez Sep 10, 2020
3529605
records fix indent
allisonsuarez Sep 10, 2020
d9081c0
changed name of test
allisonsuarez Sep 10, 2020
ba7c7e3
changed more table names
allisonsuarez Sep 10, 2020
e92afac
more whitelist fixes
allisonsuarez Sep 10, 2020
bf9e839
fixed tag logic
allisonsuarez Sep 10, 2020
09a9aef
lint
allisonsuarez Sep 16, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions metadata_service/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,15 @@
from flask_restful import Api

from metadata_service.api.column import ColumnDescriptionAPI
from metadata_service.api.dashboard import DashboardDetailAPI, DashboardDescriptionAPI, DashboardTagAPI
from metadata_service.api.dashboard import (DashboardDetailAPI, DashboardDescriptionAPI,
DashboardTagAPI, DashboardBadgeAPI)
from metadata_service.api.healthcheck import healthcheck
from metadata_service.api.popular_tables import PopularTablesAPI
from metadata_service.api.system import Neo4jDetailAPI
from metadata_service.api.table \
import TableDetailAPI, TableOwnerAPI, TableTagAPI, TableDescriptionAPI, TableDashboardAPI
import TableDetailAPI, TableOwnerAPI, TableTagAPI, TableBadgeAPI, TableDescriptionAPI, TableDashboardAPI
from metadata_service.api.tag import TagAPI
from metadata_service.api.badge import BadgeAPI
from metadata_service.api.user import (UserDetailAPI, UserFollowAPI,
UserFollowsAPI, UserOwnsAPI,
UserOwnAPI, UserReadsAPI)
Expand Down Expand Up @@ -92,6 +94,8 @@ def create_app(*, config_module_class: str) -> Flask:
'/table/<path:id>/description')
api.add_resource(TableTagAPI,
'/table/<path:id>/tag/<tag>')
api.add_resource(TableBadgeAPI,
'/table/<path:id>/badge/<badge>')
api.add_resource(TableOwnerAPI,
'/table/<path:table_uri>/owner/<owner>')
api.add_resource(TableDashboardAPI,
Expand All @@ -102,6 +106,8 @@ def create_app(*, config_module_class: str) -> Flask:
'/latest_updated_ts')
api.add_resource(TagAPI,
'/tags/')
api.add_resource(BadgeAPI,
'/badges/')
api.add_resource(UserDetailAPI,
'/user',
'/user/<path:id>')
Expand All @@ -121,6 +127,8 @@ def create_app(*, config_module_class: str) -> Flask:
'/dashboard/<path:id>/description')
api.add_resource(DashboardTagAPI,
'/dashboard/<path:id>/tag/<tag>')
api.add_resource(DashboardBadgeAPI,
'/dashboard/<path:id>/badge/<badge>')
app.register_blueprint(api_bp)

if app.config.get('SWAGGER_ENABLED'):
Expand Down
111 changes: 111 additions & 0 deletions metadata_service/api/badge.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
# Copyright Contributors to the Amundsen project.
# SPDX-License-Identifier: Apache-2.0

from flask_restful import Resource, fields, marshal

from http import HTTPStatus
from typing import Iterable, Union, Mapping, Tuple, Any

from flasgger import swag_from
from flask import current_app as app

from metadata_service.entity.resource_type import ResourceType
from metadata_service.entity.badge import Badge
from metadata_service.exception import NotFoundException
from metadata_service.proxy import get_proxy_client
from metadata_service.proxy.base_proxy import BaseProxy

badge_fields = {
'badge_name': fields.String,
'category': fields.String,
'badge_type': fields.String,
}

badges_fields = {
'badges': fields.List(fields.Nested(badge_fields))
}


class BadgeAPI(Resource):
def __init__(self) -> None:
self.client = get_proxy_client()
super(BadgeAPI, self).__init__()

@swag_from('swagger_doc/badge/badge_get.yml')
def get(self) -> Iterable[Union[Mapping, int, None]]:
"""
API to get all existing badges
"""
badges = self.client.get_badges()
return marshal({'badges': badges}, badges_fields), HTTPStatus.OK


class BadgeCommon:
def __init__(self, client: BaseProxy) -> None:
self.client = client

def put(self, id: str, resource_type: ResourceType,
badge_name: str,
category: str = '',
badge_type: str = '') -> Tuple[Any, HTTPStatus]:

if badge_type == '' or category == '':
return \
{'message': f'The badge {badge_name} for resource id {id} is not added successfully because '
f'category `{category}` and badge_type `{badge_type}` parameters are required '
'for badges'}, \
HTTPStatus.NOT_FOUND

# TODO check resource type is column when adding a badge of category column after
# implementing column level badges
whitelist_badges = app.config.get('WHITELIST_BADGES', [])
incomimg_badge = Badge(badge_name=badge_name,
category=category,
badge_type=badge_type)
# need to check whether the badge combination is part of the whitelist:

in_whitelist = False
for badge in whitelist_badges:
if incomimg_badge.badge_name == badge.badge_name and incomimg_badge.category == badge.category \
and incomimg_badge.badge_type == badge.badge_type:
in_whitelist = True
if not in_whitelist:
return \
{'message': f'The badge {badge_name} with category {category} badge_type {badge_type} for resource '
f'id {id} and resource_type {resource_type.name} is not added successfully because '
'this combination of values is not part of the whitelist'}, \
HTTPStatus.NOT_FOUND

try:
self.client.add_badge(id=id,
badge_name=badge_name,
category=category,
badge_type=badge_type,
resource_type=resource_type)
return {'message': f'The badge {badge_name} with category {category} and type {badge_type} was '
f'added successfully to resurce with id {id}'}, HTTPStatus.OK
except Exception as e:
return {'message': f'The badge {badge_name} with category {category}, badge type {badge_type} '
f'for resource id {id} and resource_type {resource_type.name} failed to '
'be added'}, \
HTTPStatus.NOT_FOUND

def delete(self, id: str, badge_name: str,
category: str,
badge_type: str,
resource_type: ResourceType) -> Tuple[Any, HTTPStatus]:
try:
self.client.delete_badge(id=id,
resource_type=resource_type,
badge_name=badge_name,
category=category,
badge_type=badge_type)
return \
{'message': f'The badge {badge_name} with category {category}, badge type {badge_type} for resource '
f'id {id} and resource_type {resource_type.name} was deleted successfully'}, \
HTTPStatus.OK
except NotFoundException:
return \
{'message': f'The badge {badge_name} with category {category}, badge type {badge_type} for resource '
f'id {id} and resource_type {resource_type.name} was not deleted successfully'}, \
HTTPStatus.NOT_FOUND
41 changes: 41 additions & 0 deletions metadata_service/api/dashboard.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@

from metadata_service.api import BaseAPI
from metadata_service.api.tag import TagCommon
from metadata_service.api.badge import BadgeCommon
from metadata_service.entity.dashboard_detail import DashboardSchema
from metadata_service.entity.description import DescriptionSchema
from metadata_service.entity.resource_type import ResourceType
Expand Down Expand Up @@ -74,6 +75,46 @@ def put(self, id: str) -> Iterable[Union[Mapping, int, None]]:
return {'message': 'id {} does not exist'.format(id)}, HTTPStatus.NOT_FOUND


class DashboardBadgeAPI(Resource):
"""
DashboardBadgeAPI that supports PUT and DELETE operation to add or delete badges
on Dashboard
"""
def __init__(self) -> None:
self.client = get_proxy_client()
self.parser = reqparse.RequestParser()
self.parser.add_argument('category', type=str, required=True)
self.parser.add_argument('badge_type', type=str, required=True)
super(DashboardBadgeAPI, self).__init__()

self._badge_common = BadgeCommon(client=self.client)

@swag_from('swagger_doc/badge/badge_put.yml')
def put(self, id: str, badge: str) -> Iterable[Union[Mapping, int, None]]:
args = self.parser.parse_args()

category = args.get('category', '')
badge_type = args.get('badge_type', '')

return self._badge_common.put(id=id,
resource_type=ResourceType.Dashboard,
badge_name=badge,
category=category,
badge_type=badge_type)

@swag_from('swagger_doc/badge/badge_delete.yml')
def delete(self, id: str, badge: str) -> Iterable[Union[Mapping, int, None]]:
args = self.parser.parse_args()
category = args.get('category', '')
badge_type = args.get('badge_type', '')

return self._badge_common.delete(id=id,
resource_type=ResourceType.Dashboard,
badge_name=badge,
category=category,
badge_type=badge_type)


class DashboardTagAPI(Resource):
"""
DashboardTagAPI that supports PUT and DELETE operation to add or delete tag
Expand Down
47 changes: 47 additions & 0 deletions metadata_service/api/swagger_doc/badge/badge_delete.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
Delete badges of a resource
---
tags:
- 'table'
- 'dashboard'
parameters:
- name: id
in: path
type: string
schema:
type: string
required: true
example: 'hive://gold.test_schema/test_table1'
- name: badge_name
in: path
type: string
schema:
type: string
required: true
example: 'beta'
- name: category
in: query
type: string
schema:
type: string
required: true
example: 'table_status'
- name: badge_type
in: query
type: string
schema:
type: string
required: true
example: 'neutral'
responses:
200:
description: 'The badge was deleted successfully'
content:
application/json:
schema:
$ref: '#/components/schemas/MessageResponse'
404:
description: 'Table or badge not found'
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
16 changes: 16 additions & 0 deletions metadata_service/api/swagger_doc/badge/badge_get.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
Get badges
---
tags:
- 'badge'
responses:
200:
description: 'Badges with sentiment and category'
content:
application/json:
schema:
type: object
properties:
badges:
type: array
items:
$ref: '#/components/schemas/Badge'
47 changes: 47 additions & 0 deletions metadata_service/api/swagger_doc/badge/badge_put.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
Add badge to a resource
---
tags:
- 'table'
- 'dashboard'
parameters:
- name: id
in: path
type: string
schema:
type: string
required: true
example: 'hive://gold.test_schema/test_table1'
- name: badge
in: path
type: string
schema:
type: string
required: true
example: 'test_badge'
- name: category
in: query
type: string
schema:
type: string
required: true
example: 'table_status'
- name: badge_type
in: query
type: string
schema:
type: string
required: true
example: 'positive'
responses:
200:
description: 'The badge was added successfully'
content:
application/json:
schema:
$ref: '#/components/schemas/MessageResponse'
404:
description: 'Table not found, or badge is not whitelisted'
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
13 changes: 13 additions & 0 deletions metadata_service/api/swagger_doc/template.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,19 @@ components:
tag_count:
type: integer
description: 'Count of times used'
Badge:
# TODO add column level badges later, probably will use this same object
type: object
properties:
badge_name:
type: string
description: 'name of the badge'
category:
type: string
description: 'category that badge belongs to'
badge_type:
type: string
description: 'negative, neutral, positive, or warning badge'
TableReaderFields:
type: object
properties:
Expand Down
37 changes: 37 additions & 0 deletions metadata_service/api/table.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@

from metadata_service.api import BaseAPI
from metadata_service.api.tag import TagCommon
from metadata_service.api.badge import BadgeCommon
from metadata_service.entity.resource_type import ResourceType
from metadata_service.entity.dashboard_summary import DashboardSummarySchema
from metadata_service.exception import NotFoundException
Expand Down Expand Up @@ -160,6 +161,42 @@ def delete(self, id: str, tag: str) -> Iterable[Union[Mapping, int, None]]:
tag_type=tag_type)


class TableBadgeAPI(Resource):
def __init__(self) -> None:
self.client = get_proxy_client()
self.parser = reqparse.RequestParser()
self.parser.add_argument('category', type=str, required=True)
self.parser.add_argument('badge_type', type=str, required=True)
super(TableBadgeAPI, self).__init__()

self._badge_common = BadgeCommon(client=self.client)

@swag_from('swagger_doc/badge/badge_put.yml')
def put(self, id: str, badge: str) -> Iterable[Union[Mapping, int, None]]:
args = self.parser.parse_args()
# TODO should I have default here?
category = args.get('category', '')
badge_type = args.get('badge_type', '')

return self._badge_common.put(id=id,
resource_type=ResourceType.Table,
badge_name=badge,
category=category,
badge_type=badge_type)

@swag_from('swagger_doc/badge/badge_delete.yml')
def delete(self, id: str, badge: str) -> Iterable[Union[Mapping, int, None]]:
args = self.parser.parse_args()
category = args.get('category', '')
badge_type = args.get('badge_type', '')

return self._badge_common.delete(id=id,
resource_type=ResourceType.Table,
badge_name=badge,
category=category,
badge_type=badge_type)


class TableDashboardAPI(BaseAPI):
"""
TableDashboard API that supports GET operation providing list of Dashboards using a table.
Expand Down
Loading