From 307c06b358ea504edbcf1cbdbfae21c0b7d3cc96 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Micka=C3=ABl=20Misbach?= Date: Wed, 29 Jan 2025 16:37:30 +0100 Subject: [PATCH 1/2] [uss_qualifier/scenarios/netrid/common_dictionary_evaluator] Setup generic evaluator for details evaluation --- .../netrid/common_dictionary_evaluator.md | 26 +-- .../netrid/common_dictionary_evaluator.py | 176 +++++++++++------- .../astm/netrid/display_data_evaluator.py | 17 +- 3 files changed, 136 insertions(+), 83 deletions(-) diff --git a/monitoring/uss_qualifier/scenarios/astm/netrid/common_dictionary_evaluator.md b/monitoring/uss_qualifier/scenarios/astm/netrid/common_dictionary_evaluator.md index 0bc9670d54..2e038a03b6 100644 --- a/monitoring/uss_qualifier/scenarios/astm/netrid/common_dictionary_evaluator.md +++ b/monitoring/uss_qualifier/scenarios/astm/netrid/common_dictionary_evaluator.md @@ -3,18 +3,18 @@ The `RIDCommonDictionaryEvaluator` only implements checks that are the responsibility of SPs and DPs. Validation or checks that are the responsibility of test designers or uss_qualifier developers are to be implemented in the calling test scenario. -## UA type +## Cases overview Note: C1, C2 & C4 are assumed to be implemented upstream of `RIDCommonDictionaryEvaluator`. -| C | Data to inject | Injection API | SP API | Observation API | Responsible entity | Failure | Failed requirement | -|----|-----------------|-----------------------------------------------------------------|--------------------------------------------------|--------------------------------------------|--------------------|-------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------| -| 1 | [invalid] | [any] | [any] | [any] | Test designer | Invalid test data provided | Exception raised in scenario bootstrap | -| 2 | [none or valid] | [anything that doesn't match data to inject, including invalid] | [any] | [any] | uss_qualifier dev | Test data not injected accurately | `interuss.automated_testing.rid.injection.UpsertTestResult` | -| 3 | [none or valid] | [matching data to inject] | [none] | [any] | SP | API-required field not provided | `astm.f3411.v22a.NET0710,X`; `astm.f3411.v22a.NET0260,Table1,X` | -| 4 | [none] | NotDeclared | [any] | [any] | uss_qualifier dev | Test data not injected accurately | `interuss.automated_testing.rid.injection.UpsertTestResult` | -| 5 | [none or valid] | [matching data to inject] | [invalid] | [any] | SP | SP API contract violated | `astm.f3411.v22a.NET0710,X`; `astm.f3411.v22a.NET0260,Table1,X` | -| 6 | [none] | [none] | [anything other than NotDeclared] | [any] | SP | UA type not communicated correctly | `astm.f3411.v22a.NET0260,Table1,X` | -| 7 | [none or valid] | [matching data to inject] | [valid, but not corresponding to injected value] | [any] | SP | UA type not communicated correctly | `astm.f3411.v22a.NET0260,Table1,X` | -| 8 | [none or valid] | [matching data to inject] | [valid] | [none] | N/A | No failure; reporting UA type to observers is not required by API | N/A | -| 9 | [none or valid] | [matching data to inject] | [valid] | [invalid] | DP | Observation API contract violated | `interuss.automated_testing.rid.observation.ObservationSuccess`; `astm.f3411.v22a.NET0450`; `astm.f3411.v22a.NET0470,Table1,X` | -| 10 | [none or valid] | [matching data to inject] | [valid] | [valid, but not corresponding to SP value] | DP | SP information incorrectly reported to observer | `astm.f3411.v22a.NET0450`; `astm.f3411.v22a.NET0470,Table1,X` | +| C | Data to inject | Injection API | SP API | Observation API | Responsible entity | Failure | Failed requirement | +|----|-----------------|-----------------------------------------------------------------|--------------------------------------------------|--------------------------------------------|--------------------|-----------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------| +| 1 | [invalid] | [any] | [any] | [any] | Test designer | Invalid test data provided | Exception raised in scenario bootstrap | +| 2 | [none or valid] | [anything that doesn't match data to inject, including invalid] | [any] | [any] | uss_qualifier dev | Test data not injected accurately | `interuss.automated_testing.rid.injection.UpsertTestResult` | +| 3 | [none or valid] | [matching data to inject] | [none] | [any] | SP | API-required field not provided | `astm.f3411.v22a.NET0710,X`; `astm.f3411.v22a.NET0260,Table1,X` | +| 4 | [none] | [Default Value] | [any] | [any] | uss_qualifier dev | Test data not injected accurately | `interuss.automated_testing.rid.injection.UpsertTestResult` | +| 5 | [none or valid] | [matching data to inject] | [invalid] | [any] | SP | SP API contract violated | `astm.f3411.v22a.NET0710,X`; `astm.f3411.v22a.NET0260,Table1,X` | +| 6 | [none] | [none] | [anything other than Default Value] | [any] | SP | Value not communicated correctly | `astm.f3411.v22a.NET0260,Table1,X` | +| 7 | [none or valid] | [matching data to inject] | [valid, but not corresponding to injected value] | [any] | SP | Value not communicated correctly | `astm.f3411.v22a.NET0260,Table1,X` | +| 8 | [none or valid] | [matching data to inject] | [valid] | [none] | N/A | No failure; reporting value to observers is not required by API | N/A | +| 9 | [none or valid] | [matching data to inject] | [valid] | [invalid] | DP | Observation API contract violated | `interuss.automated_testing.rid.observation.ObservationSuccess`; `astm.f3411.v22a.NET0450`; `astm.f3411.v22a.NET0470,Table1,X` | +| 10 | [none or valid] | [matching data to inject] | [valid] | [valid, but not corresponding to SP value] | DP | SP information incorrectly reported to observer | `astm.f3411.v22a.NET0450`; `astm.f3411.v22a.NET0470,Table1,X` | diff --git a/monitoring/uss_qualifier/scenarios/astm/netrid/common_dictionary_evaluator.py b/monitoring/uss_qualifier/scenarios/astm/netrid/common_dictionary_evaluator.py index beaa6794c0..53ba9ea587 100644 --- a/monitoring/uss_qualifier/scenarios/astm/netrid/common_dictionary_evaluator.py +++ b/monitoring/uss_qualifier/scenarios/astm/netrid/common_dictionary_evaluator.py @@ -1,13 +1,17 @@ import datetime import math -from typing import List, Optional, TypeVar, Any +from typing import List, Optional, TypeVar, Any, Union from collections.abc import Callable from arrow import ParserError from implicitdict import StringBasedDateTime from uas_standards.ansi_cta_2063_a import SerialNumber from uas_standards.astm.f3411 import v22a -from uas_standards.astm.f3411.v22a.api import UASID +from uas_standards.astm.f3411.v22a.api import ( + UASID, + UAClassificationEUCategory, + UAClassificationEUClass, +) from uas_standards.astm.f3411.v22a.constants import ( SpecialSpeed, MaxSpeed, @@ -39,7 +43,6 @@ from monitoring.uss_qualifier.configurations.configuration import ParticipantID from monitoring.uss_qualifier.resources.netrid.evaluation import EvaluationConfiguration from monitoring.uss_qualifier.scenarios.scenario import TestScenarioType - from monitoring.uss_qualifier.scenarios.scenario import ( PendingCheck, ) @@ -55,8 +58,10 @@ class RIDCommonDictionaryEvaluator(object): - generics_evaluators = [ + flight_evaluators = [ "_evaluate_ua_type", + ] + telemetry_evaluators = [ "_evaluate_timestamp_accuracy", "_evaluate_alt", "_evaluate_accuracy_v", @@ -64,6 +69,7 @@ class RIDCommonDictionaryEvaluator(object): "_evaluate_speed_accuracy", "_evaluate_vertical_speed", ] + details_evaluators = [] def __init__( self, @@ -85,12 +91,19 @@ def evaluate_sp_flight( ): """Implements fragment documented in `common_dictionary_evaluator_sp_flight.md`.""" - for generics_evaluator in self.generics_evaluators: + for generics_evaluator in self.flight_evaluators: + getattr(self, generics_evaluator)( + injected=injected_flight, + sp_observed=observed_flight, + dp_observed=None, + participant=participant_id, + query_timestamp=query_timestamp, + ) + for generics_evaluator in self.telemetry_evaluators: getattr(self, generics_evaluator)( - injected_telemetry=injected_telemetry, - injected_flight=injected_flight, - sp_observed_flight=observed_flight, - dp_observed_flight=None, + injected=injected_telemetry, + sp_observed=observed_flight, + dp_observed=None, participant=participant_id, query_timestamp=query_timestamp, ) @@ -110,12 +123,21 @@ def evaluate_dp_flight( ): """Implements fragment documented in `common_dictionary_evaluator_dp_flight.md`.""" - for generics_evaluator in self.generics_evaluators: + for generics_evaluator in self.flight_evaluators: getattr(self, generics_evaluator)( - injected_telemetry=injected_telemetry, - injected_flight=injected_flight, - sp_observed_flight=None, - dp_observed_flight=observed_flight, + injected=injected_flight, + sp_observed=None, + dp_observed=observed_flight, + participant=participants[ + 0 + ], # TODO: flatten 'participants', it always has a single value + query_timestamp=query_timestamp, + ) + for generics_evaluator in self.telemetry_evaluators: + getattr(self, generics_evaluator)( + injected=injected_telemetry, + sp_observed=None, + dp_observed=observed_flight, participant=participants[ 0 ], # TODO: flatten 'participants', it always has a single value @@ -158,29 +180,54 @@ def evaluate_dp_flight( participants, ) - def evaluate_sp_details(self, details: FlightDetails, participants: List[str]): + def evaluate_sp_details( + self, + injected_details: injection.RIDFlightDetails, + observed_details: FlightDetails, + participant_id: ParticipantID, + query_timestamp: datetime.datetime, + ): """Implements fragment documented in `common_dictionary_evaluator_sp_flight_details.md`.""" - self._evaluate_uas_id(details.raw.get("uas_id"), participants) - self._evaluate_operator_id(None, details.operator_id, participants) + for generics_evaluator in self.details_evaluators: + getattr(self, generics_evaluator)( + injected=injected_details, + sp_observed=observed_details, + dp_observed=None, + participant=participant_id, + query_timestamp=query_timestamp, + ) + + self._evaluate_uas_id(observed_details.raw.get("uas_id"), [participant_id]) + self._evaluate_operator_id(None, observed_details.operator_id, [participant_id]) self._evaluate_operator_location( None, None, None, - details.operator_location, - details.operator_altitude, - details.operator_altitude_type, - participants, + observed_details.operator_location, + observed_details.operator_altitude, + observed_details.operator_altitude_type, + [participant_id], ) def evaluate_dp_details( self, injected_details: injection.RIDFlightDetails, observed_details: Optional[observation_api.GetDetailsResponse], - participants: List[str], + participant_id: ParticipantID, + query_timestamp: datetime.datetime, ): """Implements fragment documented in `common_dictionary_evaluator_dp_flight_details.md`.""" + for generics_evaluator in self.details_evaluators: + getattr(self, generics_evaluator)( + injected=injected_details, + sp_observed=None, + dp_observed=observed_details, + participant=participant_id, + query_timestamp=query_timestamp, + ) + if not observed_details: return @@ -189,13 +236,13 @@ def evaluate_dp_details( "uas_id", injected_details.get("serial_number", None) ), # fall back on seria number if no UAS ID observed_details.get("uas", {}).get("id", None), - participants, + [participant_id], ) operator_obs = observed_details.get("operator", {}) self._evaluate_operator_id( - injected_details.operator_id, operator_obs.get("id", None), participants + injected_details.operator_id, operator_obs.get("id", None), [participant_id] ) operator_altitude_obs = operator_obs.get("altitude", {}) @@ -210,7 +257,7 @@ def evaluate_dp_details( operator_obs.get("location", None), Altitude.w84m(value=operator_altitude_value_obs), operator_altitude_obs.get("altitude_type", None), - participants, + [participant_id], ) def _evaluate_uas_id(self, value: Optional[UASID], participants: List[str]): @@ -745,7 +792,7 @@ def _evaluate_ua_type( **generic_kwargs, ): """ - Evaluates UA type. Exactly one of sp_observed_flight or dp_observed_flight must be provided. + Evaluates UA type. Exactly one of sp_observed or dp_observed must be provided. See as well `common_dictionary_evaluator.md`. Raises: @@ -796,7 +843,7 @@ def value_comparator( def _evaluate_timestamp_accuracy(self, **generic_kwargs): """ - Evaluates Timestamp accuracy. Exactly one of sp_observed_flight or dp_observed_flight must be provided. + Evaluates Timestamp accuracy. Exactly one of sp_observed or dp_observed must be provided. See as well `common_dictionary_evaluator.md`. Raises: @@ -818,7 +865,7 @@ def value_comparator(v1: Optional[float], v2: Optional[float]) -> bool: return abs(v1 - v2) < TIMESTAMP_ACCURACY_PRECISION self._generic_evaluator( - "telemetry.timestamp_accuracy", + "timestamp_accuracy", "raw.current_state.timestamp_accuracy", "current_state.timestamp_accuracy", "Timestamp accuracy", @@ -832,7 +879,7 @@ def value_comparator(v1: Optional[float], v2: Optional[float]) -> bool: def _evaluate_alt(self, **generic_kwargs): """ - Evaluates Geodetic Altitude. Exactly one of sp_observed_flight or dp_observed_flight must be provided. + Evaluates Geodetic Altitude. Exactly one of sp_observed or dp_observed must be provided. See as well `common_dictionary_evaluator.md`. Raises: @@ -847,7 +894,7 @@ def value_comparator(v1: Optional[float], v2: Optional[float]) -> bool: return abs(v1 - v2) < DISTANCE_TOLERANCE_M self._generic_evaluator( - "telemetry.position.alt", + "position.alt", "raw.current_state.position.alt", "most_recent_position.alt", "Geodetic Altitude", @@ -861,7 +908,7 @@ def value_comparator(v1: Optional[float], v2: Optional[float]) -> bool: def _evaluate_accuracy_v(self, **generic_kwargs): """ - Evaluates Geodetic Vertical Accuracy. Exactly one of sp_observed_flight or dp_observed_flight must be provided. + Evaluates Geodetic Vertical Accuracy. Exactly one of sp_observed or dp_observed must be provided. See as well `common_dictionary_evaluator.md`. Raises: @@ -881,7 +928,7 @@ def value_comparator( return v1 == v2 self._generic_evaluator( - "telemetry.position.accuracy_v", + "position.accuracy_v", "raw.current_state.position.accuracy_v", "most_recent_position.accuracy_v", "Geodetic Vertical Accuracy", @@ -915,7 +962,7 @@ def value_comparator( return v1 == v2 self._generic_evaluator( - "telemetry.position.accuracy_h", + "position.accuracy_h", "raw.current_state.position.accuracy_h", "most_recent_position.accuracy_h", "Horizontal Accuracy", @@ -929,7 +976,7 @@ def value_comparator( def _evaluate_speed_accuracy(self, **generic_kwargs): """ - Evaluates Speed Accuracy. Exactly one of sp_observed_flight or dp_observed_flight must be provided. + Evaluates Speed Accuracy. Exactly one of sp_observed or dp_observed must be provided. See as well `common_dictionary_evaluator.md`. Raises: @@ -949,7 +996,7 @@ def value_comparator( return v1 == v2 self._generic_evaluator( - "telemetry.speed_accuracy", + "speed_accuracy", "raw.current_state.speed_accuracy", "current_state.speed_accuracy", "Speed Accuracy", @@ -991,7 +1038,7 @@ def value_comparator(v1: Optional[float], v2: Optional[float]) -> bool: return abs(v1 - v2) < VERTICAL_SPEED_PRECISION self._generic_evaluator( - "telemetry.vertical_speed", + "vertical_speed", "raw.current_state.vertical_speed", "current_state.vertical_speed", "Vertical Speed", @@ -1014,17 +1061,22 @@ def _generic_evaluator( injection_required_field: bool, unknown_value: Optional[T], value_comparator: Callable[[Optional[T], Optional[T]], bool], - injected_telemetry: injection.RIDAircraftState, - injected_flight: injection.TestFlight, - sp_observed_flight: Optional[Flight], - dp_observed_flight: Optional[observation_api.Flight], + injected: Union[ + injection.TestFlight, + injection.RIDAircraftState, + injection.RIDFlightDetails, + ], + sp_observed: Optional[Union[Flight, FlightDetails]], + dp_observed: Optional[ + Union[observation_api.Flight, observation_api.GetDetailsResponse] + ], participant: ParticipantID, query_timestamp: datetime.datetime, ): """ - Generic evaluator of a field. Exactly one of sp_observed_flight or dp_observed_flight must be provided. - See as well `common_dictionary_evaluator.md`. + Generic evaluator of a field. Exactly one of sp_observed or dp_observed must be provided. + See as well `common_dictionary_evaluator.md` for an overview of the different cases 'Cn' referred to in this function. Args: injected_field_name: The name of the field on the injected flight object to test. If starts with telemetry, current telemetry is used @@ -1032,14 +1084,13 @@ def _generic_evaluator( dp_field_name: The name of the field on the dp observed flight object to test field_human_name: The display name of the field to test value_validator: If not None, pass values through this function. You may raise ValueError to indicate errors. - observed_value_validator: If not None, will be called with check and observed value, for additionnal verifications + observed_value_validator: If not None, will be called with check and observed value, for additional verifications injection_required_field: Boolean to indicate we need to check the case where nothing has been injected (C6) unknown_value: The default value that needs to be returned when nothing has been injected - value_comparator: Function that need to return True if both paramaters are equal - injected_telemetry: injected telemetry - injected_flight: injected flight as returned by the injection API. - sp_observed_flight: flight observed through the SP API. - dp_observed_flight: flight observed through the observation API. + value_comparator: Function that need to return True if both parameters are equal + injected: injected data (flight, telemetry or details). + sp_observed: flight (or details) observed through the SP API. + dp_observed: flight (or details) observed through the observation API. participant: participant providing the API through which the value was observed. query_timestamp: timestamp of the observation query. @@ -1048,24 +1099,17 @@ def _generic_evaluator( """ def dotted_get(obj: Any, key: str) -> Optional[T]: - val: Any = obj - for k in key.split("."): - val = val.get(k) - return val - - def dotted_getter(obj: Flight, key: str) -> Optional[T]: val: Any = obj for k in key.split("."): if val is None: return val - val = getattr(val, k) + if k in val: + val = val[k] + else: + val = getattr(val, k) return val - injected_val: Optional[T] = ( - dotted_get(injected_telemetry, injected_field_name[len("telemetry.") :]) - if injected_field_name.startswith("telemetry.") - else dotted_get(injected_flight, injected_field_name) - ) + injected_val: Optional[T] = dotted_get(injected, injected_field_name) if injected_val is not None: if value_validator is not None: @@ -1077,10 +1121,10 @@ def dotted_getter(obj: Flight, key: str) -> Optional[T]: ) observed_val: Optional[T] - if sp_observed_flight is not None: - observed_val = dotted_getter(sp_observed_flight, sp_field_name) - elif dp_observed_flight is not None: - observed_val = dotted_get(dp_observed_flight, dp_field_name) + if sp_observed is not None: + observed_val = dotted_get(sp_observed, sp_field_name) + elif dp_observed is not None: + observed_val = dotted_get(dp_observed, dp_field_name) else: raise ValueError("No observed flight provided.") @@ -1088,7 +1132,7 @@ def dotted_getter(obj: Flight, key: str) -> Optional[T]: f"{field_human_name} is exposed correctly", participant, ) as check: - if sp_observed_flight is not None: + if sp_observed is not None: if observed_val is None: # C3 check.record_failed( f"{field_human_name} is missing", @@ -1115,7 +1159,7 @@ def dotted_getter(obj: Flight, key: str) -> Optional[T]: participant, ) as check: - if dp_observed_flight is not None and observed_val is None: + if dp_observed is not None and observed_val is None: pass # C8 elif injected_val is None: diff --git a/monitoring/uss_qualifier/scenarios/astm/netrid/display_data_evaluator.py b/monitoring/uss_qualifier/scenarios/astm/netrid/display_data_evaluator.py index f165a42892..2cc2728c4a 100644 --- a/monitoring/uss_qualifier/scenarios/astm/netrid/display_data_evaluator.py +++ b/monitoring/uss_qualifier/scenarios/astm/netrid/display_data_evaluator.py @@ -456,9 +456,8 @@ def _evaluate_normal_observation( self._common_dictionary_evaluator.evaluate_dp_details( details_inj, details_obs, - participants=[ - observer.participant_id, - ], + observer.participant_id, + query.request.timestamp, ) def _evaluate_area_too_large_observation( @@ -993,8 +992,18 @@ def _evaluate_normal_sp_observation( query_timestamps=[details_query.query.request.timestamp], ) + # Get details that are expected to be valid for the present telemetry: + telemetry_inj = mapping.injected_flight.flight.telemetry[ + mapping.telemetry_index + ] + details_inj = mapping.injected_flight.flight.get_details( + telemetry_inj.timestamp.datetime + ) self._common_dictionary_evaluator.evaluate_sp_details( - details_query.details, [mapping.injected_flight.uss_participant_id] + details_inj, + details_query.details, + mapping.injected_flight.uss_participant_id, + details_query.query.request.timestamp, ) def _evaluate_area_too_large_sp_observation( From 377fa93ec2c05e0afbe7ccef126ab2412e748c59 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Micka=C3=ABl=20Misbach?= Date: Wed, 29 Jan 2025 16:43:52 +0100 Subject: [PATCH 2/2] [uss_qualifier/scenarios/netrid/nominal_behavior] Add checks for UA classification in DP and SP (NET0470/NET0260) --- .../netrid/common_dictionary_evaluator.py | 161 +++++++++++++++++- ..._dictionary_evaluator_dp_flight_details.md | 32 ++++ ..._dictionary_evaluator_sp_flight_details.md | 27 +++ .../suites/astm/netrid/f3411_22a.md | 22 ++- .../suites/uspace/network_identification.md | 22 ++- .../suites/uspace/required_services.md | 22 ++- 6 files changed, 282 insertions(+), 4 deletions(-) diff --git a/monitoring/uss_qualifier/scenarios/astm/netrid/common_dictionary_evaluator.py b/monitoring/uss_qualifier/scenarios/astm/netrid/common_dictionary_evaluator.py index 53ba9ea587..c3467911b7 100644 --- a/monitoring/uss_qualifier/scenarios/astm/netrid/common_dictionary_evaluator.py +++ b/monitoring/uss_qualifier/scenarios/astm/netrid/common_dictionary_evaluator.py @@ -69,7 +69,11 @@ class RIDCommonDictionaryEvaluator(object): "_evaluate_speed_accuracy", "_evaluate_vertical_speed", ] - details_evaluators = [] + details_evaluators = [ + "_evaluate_ua_classification", + "_evaluate_ua_classification_eu_category", + "_evaluate_ua_classification_eu_class", + ] def __init__( self, @@ -1050,6 +1054,161 @@ def value_comparator(v1: Optional[float], v2: Optional[float]) -> bool: **generic_kwargs, ) + def _evaluate_ua_classification( + self, + injected: injection.RIDFlightDetails, + sp_observed: Optional[FlightDetails], + dp_observed: Optional[observation_api.GetDetailsResponse], + participant: ParticipantID, + query_timestamp: datetime.datetime, + ): + """ + Evaluates UA classification type. Exactly one of sp_observed or dp_observed must be provided. + See as well `common_dictionary_evaluator.md`. + Note that the classification type is defined implicitly by presence of field 'eu_classification' or not: + > When this field is specified, the Classification Type is "European Union". If no other classification + > field is specified, the Classification Type is "Undeclared". + + Raises: + ValueError: if a test operation wasn't performed correctly by uss_qualifier. + """ + + if self._rid_version == RIDVersion.f3411_19: + self._test_scenario.record_note( + key="skip_reason", + message=f"Unsupported version {self._rid_version}: skipping UA classification type evaluation", + ) + return + + injected_ua_classification: Optional[str] = None + if "eu_classification" in injected: + injected_ua_classification = "eu_classification" + + observed_ua_classification: Optional[str] = None + if sp_observed is not None: + if sp_observed.raw.has_field_with_value("eu_classification"): + observed_ua_classification = "eu_classification" + elif dp_observed is not None: + if dp_observed.has_field_with_value( + "uas" + ) and dp_observed.uas.has_field_with_value("eu_classification"): + observed_ua_classification = "eu_classification" + else: + raise ValueError("No observed flight provided.") + + with self._test_scenario.check( + "UA classification type is consistent with injected value", + participant, + ) as check: + if dp_observed is not None and observed_ua_classification is None: + pass # C8 + + elif injected_ua_classification != observed_ua_classification: # C7 / C10 + check.record_failed( + "UA classification type is inconsistent with injected value.", + details=f"USS returned UA classification type {observed_ua_classification} yet the type injected was {injected_ua_classification}.", + query_timestamps=[query_timestamp], + ) + + def _evaluate_ua_classification_eu_category( + self, injected: injection.RIDFlightDetails, **generic_kwargs + ): + """ + Evaluates UA classification 'category' field for 'European Union' type. Exactly one of sp_observed or dp_observed must be provided. + See as well `common_dictionary_evaluator.md`. + + Raises: + ValueError: if a test operation wasn't performed correctly by uss_qualifier. + """ + + if self._rid_version == RIDVersion.f3411_19: + self._test_scenario.record_note( + key="skip_reason", + message=f"Unsupported version {self._rid_version}: skipping UA classification 'category' field for 'European Union' type", + ) + return + if not injected.has_field_with_value("eu_classification"): + # skip if UA classification type is not 'European Union' type + return + + def cat_value_validator( + val: UAClassificationEUCategory, + ) -> UAClassificationEUCategory: + return UAClassificationEUCategory(val) + + def cat_value_comparator( + v1: Optional[UAClassificationEUCategory], + v2: Optional[UAClassificationEUCategory], + ) -> bool: + + if v1 is None or v2 is None: + return False + + return v1 == v2 + + self._generic_evaluator( + "eu_classification.category", + "raw.eu_classification.category", + "uas.eu_classification.category", + "UA classification 'category' field for 'European Union' type", + cat_value_validator, + None, + False, + UAClassificationEUCategory.EUCategoryUndefined, + cat_value_comparator, + injected, + **generic_kwargs, + ) + + def _evaluate_ua_classification_eu_class( + self, injected: injection.RIDFlightDetails, **generic_kwargs + ): + """ + Evaluates UA classification 'class' field for 'European Union' type. Exactly one of sp_observed or dp_observed must be provided. + See as well `common_dictionary_evaluator.md`. + + Raises: + ValueError: if a test operation wasn't performed correctly by uss_qualifier. + """ + + if self._rid_version == RIDVersion.f3411_19: + self._test_scenario.record_note( + key="skip_reason", + message=f"Unsupported version {self._rid_version}: skipping UA classification 'class' field for 'European Union' type", + ) + return + if not injected.has_field_with_value("eu_classification"): + # skip if UA classification type is not 'European Union' type + return + + def class_value_validator( + val: UAClassificationEUClass, + ) -> UAClassificationEUClass: + return UAClassificationEUClass(val) + + def class_value_comparator( + v1: Optional[UAClassificationEUClass], v2: Optional[UAClassificationEUClass] + ) -> bool: + + if v1 is None or v2 is None: + return False + + return v1 == v2 + + self._generic_evaluator( + "eu_classification.class", + "raw.eu_classification.class", + "uas.eu_classification.class", + "UA classification 'class' field for 'European Union' type", + class_value_validator, + None, + False, + UAClassificationEUClass.EUClassUndefined, + class_value_comparator, + injected, + **generic_kwargs, + ) + def _generic_evaluator( self, injected_field_name: str, diff --git a/monitoring/uss_qualifier/scenarios/astm/netrid/v22a/common_dictionary_evaluator_dp_flight_details.md b/monitoring/uss_qualifier/scenarios/astm/netrid/v22a/common_dictionary_evaluator_dp_flight_details.md index b51e2736c6..7237bd3ac4 100644 --- a/monitoring/uss_qualifier/scenarios/astm/netrid/v22a/common_dictionary_evaluator_dp_flight_details.md +++ b/monitoring/uss_qualifier/scenarios/astm/netrid/v22a/common_dictionary_evaluator_dp_flight_details.md @@ -14,6 +14,38 @@ This fragment is implemented in `common_dictionary_evaluator.py:RIDCommonDiction If the UAS ID contained in flight details returned by a display provider does not correspond to the injected one, the DP is not providing accurate data and is thus in breach of **[astm.f3411.v22a.NET0450](../../../../requirements/astm/f3411/v22a.md)** +## ⚠️ UA classification type is consistent with injected value check + +If the UA classification type value exposed by the observer API is inconsistent with the injected value this check will fail per: +**[astm.f3411.v22a.NET0450](../../../../requirements/astm/f3411/v22a.md)** because the DP fails to provide accurate data; +**[astm.f3411.v22a.NET0470,Table1,4](../../../../requirements/astm/f3411/v22a.md)** because the DP fails to expose data consistent with the valid injected value. + +## ⚠️ UA classification 'category' field for 'European Union' type is exposed correctly check + +If the UA classification 'category' field for 'European Union' type value exposed by the observation API is invalid this check will fail per: +**[interuss.automated_testing.rid.observation.ObservationSuccess](../../../../requirements/interuss/automated_testing/rid/observation.md)** because the DP violates the observation API contract; +**[astm.f3411.v22a.NET0450](../../../../requirements/astm/f3411/v22a.md)** because the DP fails to provide accurate data; +**[astm.f3411.v22a.NET0470,Table1,3](../../../../requirements/astm/f3411/v22a.md)** because the DP fails to expose data consistent with the Common Data Dictionary. + +## ⚠️ UA classification 'category' field for 'European Union' type is consistent with injected value check + +If the UA classification 'category' field for 'European Union' type value exposed by the observer API is inconsistent with the injected value this check will fail per: +**[astm.f3411.v22a.NET0450](../../../../requirements/astm/f3411/v22a.md)** because the DP fails to provide accurate data; +**[astm.f3411.v22a.NET0470,Table1,3](../../../../requirements/astm/f3411/v22a.md)** because the DP fails to expose data consistent with the valid injected value. + +## ⚠️ UA classification 'class' field for 'European Union' type is exposed correctly check + +If the UA classification 'class' field for 'European Union' type value exposed by the observation API is invalid this check will fail per: +**[interuss.automated_testing.rid.observation.ObservationSuccess](../../../../requirements/interuss/automated_testing/rid/observation.md)** because the DP violates the observation API contract; +**[astm.f3411.v22a.NET0450](../../../../requirements/astm/f3411/v22a.md)** because the DP fails to provide accurate data; +**[astm.f3411.v22a.NET0470,Table1,3](../../../../requirements/astm/f3411/v22a.md)** because the DP fails to expose data consistent with the Common Data Dictionary. + +## ⚠️ UA classification 'class' field for 'European Union' type is consistent with injected value check + +If the UA classification 'class' field for 'European Union' type value exposed by the observer API is inconsistent with the injected value this check will fail per: +**[astm.f3411.v22a.NET0450](../../../../requirements/astm/f3411/v22a.md)** because the DP fails to provide accurate data; +**[astm.f3411.v22a.NET0470,Table1,3](../../../../requirements/astm/f3411/v22a.md)** because the DP fails to expose data consistent with the valid injected value. + ## Operator ID consistency with Common Dictionary check **[astm.f3411.v22a.NET0470](../../../../requirements/astm/f3411/v22a.md)** requires that Net-RID Display Provider shall (NET0470) provide access to required and optional fields to Remote ID Display Applications according to the Common Dictionary. This check validates that the Operator ID, if present, is valid. (**[astm.f3411.v22a.NET0470,Table1,9](../../../../requirements/astm/f3411/v22a.md)**) diff --git a/monitoring/uss_qualifier/scenarios/astm/netrid/v22a/common_dictionary_evaluator_sp_flight_details.md b/monitoring/uss_qualifier/scenarios/astm/netrid/v22a/common_dictionary_evaluator_sp_flight_details.md index baecd7db4b..05e21bd39b 100644 --- a/monitoring/uss_qualifier/scenarios/astm/netrid/v22a/common_dictionary_evaluator_sp_flight_details.md +++ b/monitoring/uss_qualifier/scenarios/astm/netrid/v22a/common_dictionary_evaluator_sp_flight_details.md @@ -10,6 +10,33 @@ NET0260 requires that relevant Remote ID data, consistent with the common data d NET0260 requires that relevant Remote ID data, consistent with the common data dictionary, be reported by the Service Provider. This check validates that the UAS ID is in serial number format. (**[astm.f3411.v22a.NET0260,Table1,1a](../../../../requirements/astm/f3411/v22a.md)**) +## ⚠️ UA classification type is consistent with injected value check + +If the UA classification type value exposed by the SP API is inconsistent with the injected value this check will fail per: +**[astm.f3411.v22a.NET0260,Table1,4](../../../../requirements/astm/f3411/v22a.md)** because the SP fails to expose data consistent with the valid injected value. + +## ⚠️ UA classification 'category' field for 'European Union' type is exposed correctly check + +If the UA classification 'category' field for 'European Union' type value exposed by the SP API is missing or invalid this check will fail per: +**[astm.f3411.v22a.NET0710,2](../../../../requirements/astm/f3411/v22a.md)** because the SP violates the SP API contract; +**[astm.f3411.v22a.NET0260,Table1,3](../../../../requirements/astm/f3411/v22a.md)** because the SP fails to expose data consistent with the Common Data Dictionary. + +## ⚠️ UA classification 'category' field for 'European Union' type is consistent with injected value check + +If the UA classification 'category' field for 'European Union' type value exposed by the SP API is inconsistent with the injected value this check will fail per: +**[astm.f3411.v22a.NET0260,Table1,3](../../../../requirements/astm/f3411/v22a.md)** because the SP fails to expose data consistent with the valid injected value. + +## ⚠️ UA classification 'class' field for 'European Union' type is exposed correctly check + +If the UA classification 'class' field for 'European Union' type value exposed by the SP API is missing or invalid this check will fail per: +**[astm.f3411.v22a.NET0710,2](../../../../requirements/astm/f3411/v22a.md)** because the SP violates the SP API contract; +**[astm.f3411.v22a.NET0260,Table1,3](../../../../requirements/astm/f3411/v22a.md)** because the SP fails to expose data consistent with the Common Data Dictionary. + +## ⚠️ UA classification 'class' field for 'European Union' type is consistent with injected value check + +If the UA classification 'class' field for 'European Union' type value exposed by the SP API is inconsistent with the injected value this check will fail per: +**[astm.f3411.v22a.NET0260,Table1,3](../../../../requirements/astm/f3411/v22a.md)** because the SP fails to expose data consistent with the valid injected value. + ## Operator ID consistency with Common Dictionary check NET0260 requires that relevant Remote ID data, consistent with the common data dictionary, be reported by the Service Provider. This check validates that the Operator ID, if present, is expressed as ASCII text. (**[astm.f3411.v22a.NET0260,Table1,9](../../../../requirements/astm/f3411/v22a.md)**) diff --git a/monitoring/uss_qualifier/suites/astm/netrid/f3411_22a.md b/monitoring/uss_qualifier/suites/astm/netrid/f3411_22a.md index 0853863917..97ac058e90 100644 --- a/monitoring/uss_qualifier/suites/astm/netrid/f3411_22a.md +++ b/monitoring/uss_qualifier/suites/astm/netrid/f3411_22a.md @@ -24,7 +24,7 @@ Checked in - astm
.f3411
.v22a + astm
.f3411
.v22a DSS0010 Implemented ASTM NetRID DSS: Token Validation @@ -359,6 +359,16 @@ Implemented ASTM NetRID nominal behavior + + NET0260,Table1,3 + Implemented + ASTM NetRID nominal behavior + + + NET0260,Table1,4 + Implemented + ASTM NetRID nominal behavior + NET0260,Table1,6 Implemented @@ -514,6 +524,16 @@ Implemented ASTM NetRID nominal behavior + + NET0470,Table1,3 + Implemented + ASTM NetRID nominal behavior + + + NET0470,Table1,4 + Implemented + ASTM NetRID nominal behavior + NET0470,Table1,5 Implemented diff --git a/monitoring/uss_qualifier/suites/uspace/network_identification.md b/monitoring/uss_qualifier/suites/uspace/network_identification.md index 8cc5580a36..a68111d1bf 100644 --- a/monitoring/uss_qualifier/suites/uspace/network_identification.md +++ b/monitoring/uss_qualifier/suites/uspace/network_identification.md @@ -17,7 +17,7 @@ Checked in - astm
.f3411
.v22a + astm
.f3411
.v22a DSS0010 Implemented ASTM NetRID DSS: Token Validation @@ -352,6 +352,16 @@ Implemented ASTM NetRID nominal behavior + + NET0260,Table1,3 + Implemented + ASTM NetRID nominal behavior + + + NET0260,Table1,4 + Implemented + ASTM NetRID nominal behavior + NET0260,Table1,6 Implemented @@ -507,6 +517,16 @@ Implemented ASTM NetRID nominal behavior + + NET0470,Table1,3 + Implemented + ASTM NetRID nominal behavior + + + NET0470,Table1,4 + Implemented + ASTM NetRID nominal behavior + NET0470,Table1,5 Implemented diff --git a/monitoring/uss_qualifier/suites/uspace/required_services.md b/monitoring/uss_qualifier/suites/uspace/required_services.md index 672c30dc6b..9202ca89af 100644 --- a/monitoring/uss_qualifier/suites/uspace/required_services.md +++ b/monitoring/uss_qualifier/suites/uspace/required_services.md @@ -18,7 +18,7 @@ Checked in - astm
.f3411
.v22a + astm
.f3411
.v22a DSS0010 Implemented ASTM NetRID DSS: Token Validation @@ -353,6 +353,16 @@ Implemented ASTM NetRID nominal behavior + + NET0260,Table1,3 + Implemented + ASTM NetRID nominal behavior + + + NET0260,Table1,4 + Implemented + ASTM NetRID nominal behavior + NET0260,Table1,6 Implemented @@ -508,6 +518,16 @@ Implemented ASTM NetRID nominal behavior + + NET0470,Table1,3 + Implemented + ASTM NetRID nominal behavior + + + NET0470,Table1,4 + Implemented + ASTM NetRID nominal behavior + NET0470,Table1,5 Implemented