Registry manifest and Schema diff (#400)

* feat(semconv): create registry manifest * feat(schema): Parse OTEL schema 1.1 * feat(cli): Add registry diff command (WIP) * feat(schema): Support new deprecated field format * feat(diff): Add registry diff command. * feat(diff): Add diff for metrics * feat(diff): Add diff for events, spans, and resources * feat(diff): Add template rendering capability * feat(diff): Add diff-format for ansi, ansi-stats, json, and yaml * feat(diff): Add diff-format for markdown * chore(diff): Fix unit tests * feat(cli): Prepare registry update-schema command * feat(cli): Prepare registry update-schema command * feat(cli): Prepare registry update-schema command * chore(typo): fix typo in comment * chore(build): Resolve merging with main upstream * chore(build): Clean up code to prepare PR * chore(build): Clean up code to prepare PR * chore(build): Prepare resolve_telemetry_schema to return WResult * chore(build): Fix `registry diff` to return a diff even when there are warnings * chore(merge): Merge with main branch * feat(diff): Test schema diff feature * feat(diff): Remove complex logic infering the type of deprecation based on the message * feat(diff): Rename the field Deprecated::Renamed::new_name into Deprecated::Renamed::renamed_to * chore(doc): Fix documentation of the ResolvedTelemetrySchema::groups method * chore(doc): Fix documentation based on @jsuereth feedback * chore(merge): Merge with upstream main branch * feat(diff): Documentation of the schema diff command and the underlying data model. * feat(diff): Compute a diff only on the registry attributes. * feat(diff): Use the group.id as unique identifier for the groups. * feat(diff): Update unit tests to check deprecated attributes * Update crates/weaver_resolved_schema/src/lib.rs Co-authored-by: Jeremy Blythe <[email protected]> * Update crates/weaver_resolved_schema/src/lib.rs Co-authored-by: Jeremy Blythe <[email protected]> * chore: Merge with main upstream * chore(build): Update allowed_external_types to allow classic dependencies to be part of the public API. * chore(build): Update allowed_external_types to allow classic dependencies to be part of the public API. * chore(merge): Fix merge issue with main with main branch. * feat(schema-diff): Describe various situations of schema changes. * feat(schema-diff): Describe various situations of schema changes. * feat(schema-diff): Update schema changes intro * feat(schema-diff): Update schema changes intro * chore(doc): Update document talking about schema changes * chore(doc): Update schema-changes.md * feat(diff): Update schema diff based on SIG meeting decision * feat(diff): Fix diff templates to use registry_attributes instead of attributes * feat(diff): Fix merge * feat(diff): Update the code to take into account PR feedbacks. * feat(diff): Add a todo based on PR feedback and update of the CHANGELOG.md * feat(diff): Add `is_registry_attribute_group` method in Group * feat(diff): Update deprecated and schema change event based on last feedback. * feat(diff): Improve CHANGELOG.md * feat(diff): Improve CHANGELOG.md * Update crates/weaver_semconv/src/deprecated.rs Co-authored-by: Jeremy Blythe <[email protected]> * Update schemas/semconv.schema.json Co-authored-by: Jeremy Blythe <[email protected]> * feat(diff): Fix templates and add unit tests * feat(diff): Fix fmt issues --------- Co-authored-by: Jeremy Blythe <[email protected]>
open-telemetry · Feb 4, 2025 · 5eaa384 · 5eaa384
1 parent 6147050
commit 5eaa384
Show file tree

Hide file tree

Showing 79 changed files with 4,468 additions and 325 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -6,6 +6,15 @@ All notable changes to this project will be documented in this file.
 
 What's changed
 
+* **Breaking Change**: Introduced a new `weaver registry diff` command to generate a diff report between two versions of
+  the semantic convention registry. This PR introduces a breaking change in the semantic conventions schema. While the
+  text-based `deprecated` field is still supported for compatibility reasons, future semantic conventions should use the
+  new `deprecated` structured format. ([#400](https://github.com/open-telemetry/weaver/pull/400/) by @lquerel).
+  * The `deprecated` field is now a structured field defining the precise reason for deprecation. The semantic
+    conventions must be updated to adopt this new format.
+  * The changes related to the `deprecated` field (i.e., string → struct) also have a potential impact on certain
+    templates that reference the `deprecated` field as containing text. These templates will need to be updated to use
+    the `brief` field, which provides a textual explanation of the reasons for the deprecation.
 * For issue [#569](Add include_stability config into semconv_grouped_attributes): `is_experimental` returns `true` by default. ([#570](https://github.com/open-telemetry/weaver/pull/570) by @jerbly). 
 * Added an OTLP receiver to Weaver to prepare for the `weaver registry live-check` command. (see [#548](https://github.com/open-telemetry/weaver/pull/548) by @lquerel)
 * Refactored CLI registry commands to remove some duplication. Resolving the registry with policy checks is common for `generate`, `resolve` and `check`. ([#536](https://github.com/open-telemetry/weaver/pull/536) by @jerbly).

diff --git a/Cargo.lock b/Cargo.lock
diff --git a/Cargo.toml b/Cargo.toml
@@ -79,6 +79,7 @@ weaver_semconv_gen = { path = "crates/weaver_semconv_gen" }
 weaver_cache = { path = "crates/weaver_cache" }
 weaver_forge = { path = "crates/weaver_forge" }
 weaver_checker = { path = "crates/weaver_checker" }
+weaver_otel_schema = { path = "crates/weaver_otel_schema" }
 
 clap = { version = "4.5.27", features = ["derive"] }
 rayon = "1.10.0"
@@ -105,10 +106,12 @@ log = "0.4.22"
 
 [dev-dependencies]
 weaver_diff = { path = "crates/weaver_diff" }
+weaver_version = { path = "crates/weaver_version" }
 tempdir.workspace = true
 ureq.workspace = true
 assert_cmd = "2.0.16"
 portpicker = "0.1.1"
+tempfile = "3.16.0"
 #opentelemetry_sdk = { version = "0.27.1", features = [] }
 #opentelemetry-otlp = { version ="0.27.0", features = ["metrics", "logs", "trace"] }
 

diff --git a/README.md b/README.md
@@ -147,6 +147,7 @@ Semantic Convention Registries. The following commands are available:
 |---------------------------------------------------------------------------|---------------------------------------------|
 | [weaver registry check](docs/usage.md#registry-check)                     | Check the validity of a semconv registry    |
 | [weaver registry resolve](docs/usage.md#registry-resolve)                 | Resolve a semconv registry                  |
+| [weaver registry diff](docs/usage.md#registry-diff)                       | Generate a diff report between two versions |
 | [weaver registry generate](docs/usage.md#registry-generate)               | Generate artifacts from a semconv registry  |
 | [weaver registry update-markdown](docs/usage.md#registry-update-markdown) | Update semconv snippet-based markdown files |
 | [weaver registry stats](docs/usage.md#registry-stats)                     | Generate statistics on a semconv registry   |
@@ -162,6 +163,8 @@ Telemetry Schemas.
   documentation and code based on semantic conventions.
 - [Weaver Checker](crates/weaver_checker/README.md): An integrated policy
   engine for enforcing policies on semantic conventions.
+- [Schema Changes](docs/schema-changes.md): A document describing the data model
+  used to represent the differences between two versions of a semantic convention registry.
 - [Application Telemetry Schema OTEP](https://github.com/open-telemetry/oteps/blob/main/text/0243-app-telemetry-schema-vision-roadmap.md):
   A vision and roadmap for the concept of Application Telemetry Schema.
 - Presentation slides from the Semantic Convention SIG meeting on October 23,

diff --git a/crates/weaver_cache/src/lib.rs b/crates/weaver_cache/src/lib.rs
@@ -29,6 +29,8 @@ pub mod registry_path;
 const TAR_GZ_EXT: &str = ".tar.gz";
 /// The extension for a zip archive.
 const ZIP_EXT: &str = ".zip";
+/// The name of the registry manifest file.
+pub const REGISTRY_MANIFEST: &str = "registry_manifest.yaml";
 
 /// An error that can occur while creating or using a cache.
 #[derive(thiserror::Error, Debug, Clone, Serialize, Diagnostic)]
@@ -99,7 +101,7 @@ impl From<Error> for DiagnosticMessages {
 /// - A simple wrapper around a local directory
 /// - Initialized from a Git repository
 /// - Initialized from a Git archive
-#[derive(Default)]
+#[derive(Default, Debug)]
 pub struct RegistryRepo {
     // A unique identifier for the registry (e.g. main, baseline, etc.)
     id: String,
@@ -509,6 +511,17 @@ impl RegistryRepo {
         &self.registry_path
     }
 
+    /// Returns the path to the `registry_manifest.yaml` file (if any).
+    #[must_use]
+    pub fn manifest_path(&self) -> Option<PathBuf> {
+        let manifest_path = self.path.join(REGISTRY_MANIFEST);
+        if manifest_path.exists() {
+            Some(manifest_path)
+        } else {
+            None
+        }
+    }
+
     /// Creates a temporary directory for the registry repository and returns the path.
     /// The temporary directory is created in the `.weaver/semconv_registry_cache`.
     fn create_tmp_repo() -> Result<TempDir, Error> {

diff --git a/crates/weaver_codegen_test/build.rs b/crates/weaver_codegen_test/build.rs
@@ -24,7 +24,6 @@ use weaver_semconv::registry::SemConvRegistry;
 
 const SEMCONV_REGISTRY_PATH: &str = "./semconv_registry/";
 const TEMPLATES_PATH: &str = "./templates/registry/";
-const REGISTRY_ID: &str = "test";
 const TARGET: &str = "rust";
 const FOLLOW_SYMLINKS: bool = false;
 
@@ -50,22 +49,20 @@ fn main() {
         .ignore(|e| matches!(e.severity(), Some(miette::Severity::Warning)))
         .into_result_failing_non_fatal()
         .unwrap_or_else(|e| process_error(&logger, e));
-    let mut registry = SemConvRegistry::from_semconv_specs(REGISTRY_ID, semconv_specs);
+    let mut registry = SemConvRegistry::from_semconv_specs(&registry_repo, semconv_specs)
+        .unwrap_or_else(|e| process_error(&logger, e));
     let schema = SchemaResolver::resolve_semantic_convention_registry(&mut registry)
+        .into_result_failing_non_fatal()
         .unwrap_or_else(|e| process_error(&logger, e));
 
     let loader = FileSystemFileLoader::try_new(TEMPLATES_PATH.into(), TARGET)
         .unwrap_or_else(|e| process_error(&logger, e));
     let config = WeaverConfig::try_from_path("./templates/registry/rust")
         .unwrap_or_else(|e| process_error(&logger, e));
     let engine = TemplateEngine::new(config, loader, Params::default());
-    let template_registry = ResolvedRegistry::try_from_resolved_registry(
-        schema
-            .registry(REGISTRY_ID)
-            .expect("Failed to get the registry from the resolved schema"),
-        schema.catalog(),
-    )
-    .unwrap_or_else(|e| process_error(&logger, e));
+    let template_registry =
+        ResolvedRegistry::try_from_resolved_registry(&schema.registry, schema.catalog())
+            .unwrap_or_else(|e| process_error(&logger, e));
     let target_dir: PathBuf = target_dir.into();
     engine
         .generate(

diff --git a/crates/weaver_common/src/diagnostic.rs b/crates/weaver_common/src/diagnostic.rs
@@ -75,6 +75,12 @@ pub struct DiagnosticMessage {
 #[serde(transparent)]
 pub struct DiagnosticMessages(Vec<DiagnosticMessage>);
 
+impl From<DiagnosticMessage> for DiagnosticMessages {
+    fn from(value: DiagnosticMessage) -> Self {
+        Self(vec![value])
+    }
+}
+
 impl DiagnosticMessage {
     /// Creates a new diagnostic message from an error
     pub fn new<M: Error + Diagnostic + Serialize + Send + Sync + 'static>(error: M) -> Self {
@@ -180,8 +186,7 @@ impl DiagnosticMessages {
         Self(vec![DiagnosticMessage::new(error)])
     }
 
-    /// Returns true if all the diagnostic messages are explicitly marked as
-    /// warnings or advices.
+    /// Returns true if at least one diagnostic message has an error severity.
     #[must_use]
     pub fn has_error(&self) -> bool {
         let non_error_count = self
@@ -242,7 +247,7 @@ where
 #[cfg(test)]
 mod tests {
     use super::*;
-    use miette::{diagnostic, Diagnostic};
+    use miette::Diagnostic;
 
     #[derive(thiserror::Error, Debug, Clone, Diagnostic, Serialize)]
     #[error("This is a test error")]