Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[pull] master from datahub-project:master #59

Open
wants to merge 84 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
84 commits
Select commit Hold shift + click to select a range
47134c2
feat: update ml system UI (#12334)
yoonhyejin Jan 29, 2025
22e012a
feat(ingestion/airflow): Add support for mutiple datahub emitter (#12…
treff7es Jan 29, 2025
cc0d43f
feat(ingest): add datahub apply source (#12482)
anshbansal Jan 29, 2025
412600a
feat(telemetry): cross-component async write tracing (#12405)
david-leifker Jan 29, 2025
d0af9d5
add datahub-v1.0-rc issue template
maggiehays Jan 29, 2025
a35bfdd
fix(nocode): fix no-code upgrade (#12494)
david-leifker Jan 29, 2025
280e82a
Updated DataHub v1.0-rc template
maggiehays Jan 29, 2025
c2a4358
Bring Alchemy UI to OSS (#12481)
chriscollins3456 Jan 30, 2025
ffce018
feat(sdk): data process instance - container, subtype, dataplatformin…
shirshanka Jan 30, 2025
469e56f
docs: add ai integration tutorials and sample scripts (#12487)
yoonhyejin Jan 30, 2025
bc09b28
refactor(operation_config): change logging on is_profiling_enabled (#…
cccs-cat001 Jan 30, 2025
6acd94b
feat(docs) assertion execution behavior (#12484)
jayacryl Jan 30, 2025
c8e88ae
feat(ingest/dbt-core): support fetching using the s3a protocol (#12465)
haon85 Jan 30, 2025
aaaa655
feat(dataProcessInstance): Support data process instance entity page …
asikowitz Jan 30, 2025
ddb3db9
docs: change heading of ingestion page (#12501)
anshbansal Jan 30, 2025
f527c5e
feat(iceberg-rest): implement iceberg REST catalog api (#12500)
david-leifker Jan 30, 2025
d091fff
fix(tableau): fixes missed lineage (#12434)
sgomezvillamor Jan 30, 2025
a80d23e
fix(test): fix metadata-io test (#12503)
david-leifker Jan 30, 2025
a155470
fix(ci): disable ci telemetry modelDocUpload (#12504)
david-leifker Jan 30, 2025
7a32256
docs(ingest/mode): add details on authentication/permissions for mode…
hsheth2 Jan 30, 2025
4dc9bfc
fix(ingest/snowflake): Create all structured propery templates before…
treff7es Jan 30, 2025
301d628
docs: fix token to be not required in sample script (#12511)
yoonhyejin Jan 31, 2025
a7598ca
fix(mssql): adds missing containers and browsepathsv2 for dataflow an…
sgomezvillamor Jan 31, 2025
bde5c73
fix(ingest/glue): change to warning on access denied (#12519)
anshbansal Jan 31, 2025
9f111f8
fix(ingest/mode): remove unused field (#12520)
anshbansal Jan 31, 2025
4d5f3a5
docs: fix link to executor helm chart (#12522)
anshbansal Jan 31, 2025
6d05971
fix(ingest): add missing dep for gcs (#12505)
hsheth2 Jan 31, 2025
df24dfa
docs(entity-change-events): add docs for action request events (#12493)
gabe-lyons Jan 31, 2025
e28e682
docs(ingest): script to add ERModelRelationship Entity (#12473)
sagar-salvi-apptware Jan 31, 2025
e2b1ed7
refactor(trace-model): refactor trace model package (#12510)
david-leifker Jan 31, 2025
6963863
fix(ci): run smoke tests on release (#12518)
chakru-r Jan 31, 2025
ec0eb06
chore(bump): bump jmx version (#12524)
david-leifker Jan 31, 2025
317b740
fix(cli): avoid false positive cli upgrade suggestions (#12497)
hsheth2 Jan 31, 2025
1e0f993
fix(ingest/azure-ad): limit the size of the ingestion report (#12498)
hsheth2 Jan 31, 2025
3757a44
feat(metadata-io): enable rollback transaction support (#12509)
david-leifker Jan 31, 2025
ffc98da
feat(snowflake): add missing pushdown_deny_usernames config to be use…
sgomezvillamor Feb 3, 2025
64aaaf1
fix(model): fixes DashboardContainsDashboard relationship in Dashboar…
sgomezvillamor Feb 3, 2025
6f0d475
feat(restoreIndices): update restore indices args and docs (#12529)
RyanHolstien Feb 3, 2025
d2b0e57
fix(businessAttribute): fix business Attribute related entities (#12537)
deepgarg-visa Feb 3, 2025
469cc4f
fix(ui): make data process instance visible in container in V2& fix m…
yoonhyejin Feb 3, 2025
35e2bbb
fix(ingest): avoid multiprocessing "fork" start method (#12543)
hsheth2 Feb 4, 2025
de55788
fix(ui): revert backend breaking changes to mau (#12461)
kevinkarchacryl Feb 4, 2025
6cb5191
tests(kafka-connect): fixes integration tests setup (#12531)
sgomezvillamor Feb 4, 2025
808447d
fix(ingest/unity): add row count in table profile of delta tables (#1…
mayurinehate Feb 4, 2025
d1e8a0a
fix(ingest): use lossy collections (#12523)
anshbansal Feb 4, 2025
a4f64fd
fix(misc-openapi): fix openlineage, platform events & swagger (#12539)
david-leifker Feb 4, 2025
e1ce780
fix(test): move reading env variable inside method (#12549)
anshbansal Feb 4, 2025
b51327e
feat(versioning): Add V2 UI; make backend more synchronous; add to co…
asikowitz Feb 4, 2025
7f6e399
docs(iceberg): add iceberg user guide (#12533)
chakru-r Feb 4, 2025
ac13f25
feat(ingestion/snowflake):adds streams as a new dataset with lineage …
brock-acryl Feb 5, 2025
32b654c
feat(powerbi): Report to Dashboard lineage (#12451)
sgomezvillamor Feb 5, 2025
66bce0d
fix(no-rows-updated): fix no rows updated (#12530)
david-leifker Feb 5, 2025
1cb9165
ci(smoke): report smoke test results to codecov (#12556)
hsheth2 Feb 5, 2025
468112b
feat(UI): Confirmation before deleting Link (#12162)
pinakipb2 Feb 5, 2025
5ed4b5b
feat(ingest/s3): ignore depth mismatched path (#12326)
eagle-25 Feb 5, 2025
8773ff5
feat(docs-site) adding case studies and updating banner (#12525)
jayacryl Feb 5, 2025
06bee0d
feat(ingest/mongodb) re-order aggregation logic (#12428)
Haebuk Feb 5, 2025
23a86fd
docs(salesforce): add missing salesforce source to cli doc (#12550)
remisalmon Feb 5, 2025
7f88710
feat(openapi): precondition exceptions return 412 (#12552)
david-leifker Feb 5, 2025
0ed3d7f
feat(openapi): point in time parameter (elasticsearch only) (#12553)
david-leifker Feb 6, 2025
65376ee
fix(openapi-spec): fix openapi spec oneOf schema (#12561)
david-leifker Feb 6, 2025
52f71dd
fix(autocomplete): fix autocomplete duplicate field (#12558)
david-leifker Feb 6, 2025
a98d4c2
build(deps): bump black from 23.7.0 to 24.3.0 in /metadata-service/ic…
dependabot[bot] Feb 6, 2025
4448fc2
feat(sdk): add scaffolding for sdk v2 (#12554)
hsheth2 Feb 6, 2025
4d998da
doc(dbt): Add missing dbt extra requirement to cli doc (#12568)
remisalmon Feb 6, 2025
140b71f
feat(docs): Add live secret reload in k8s remote executor page (#12541)
pedro93 Feb 7, 2025
180087f
fix(ingest): remove duplicate mcps,more typing (#12557)
mayurinehate Feb 7, 2025
bd47b11
doc: update doc of first release (#12574)
anshbansal Feb 7, 2025
9f27d59
fix(docs): need to restore indices when adding @Searchable (#12576)
jayacryl Feb 7, 2025
7c19542
fix(sdk): fix platform instance generation in the sdk (#12573)
hsheth2 Feb 7, 2025
390a672
fix(looker): sort user mapping for consistency (#12569)
hsheth2 Feb 7, 2025
45c8123
fix(ingestion/teradata): teradata profiling fix for pooling (#12507)
brock-acryl Feb 7, 2025
03f1f2d
fix(structuredProps) Add validation for allowedTypes and harden API f…
chriscollins3456 Feb 7, 2025
03bce47
fix(ui): better experience for analytics charts (#12462)
kevinkarchacryl Feb 7, 2025
32c62e5
feat(ingest/mssql): improve stored procedure splitting (#12563)
hsheth2 Feb 7, 2025
2615e07
docs: add page on metadata standards (#12584)
hsheth2 Feb 10, 2025
6a35cd6
feat(gh-workflows) adding jayacryl to pr-labeler (#12579)
jayacryl Feb 10, 2025
704526b
fix(iceberg): delete associated platform resources when deleting ware…
chakru-r Feb 10, 2025
2323aff
feat(ingest): add display name for dynamodb tables (#12534)
mayurinehate Feb 10, 2025
15ac294
fix(ui) Show editable field info for fields based on exact fieldPath …
chriscollins3456 Feb 10, 2025
6098e97
fix(openapi-schema): fix openapi schema generator (#12590)
david-leifker Feb 11, 2025
20409fd
feat(ingestion/dbt): Add include_database_name parameter for dbt core…
svdimchenko Feb 11, 2025
60bbedf
fix(web) ingestion page resets when filter updated (#12589)
jayacryl Feb 11, 2025
96022f2
dev: update pre-commit config (#12592)
anshbansal Feb 12, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/--bug-report.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,4 +29,4 @@ If applicable, add screenshots to help explain your problem.
- Version [e.g. 22]

**Additional context**
Add any other context about the problem here.
Add any other context about the problem here.
32 changes: 32 additions & 0 deletions .github/ISSUE_TEMPLATE/datahub-v1-0-rc-bug-report.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
---
name: DataHub v1.0-rc Bug Report
about: Report issues found in DataHub v1.0 Release Candidates
title: "[v1.0-rc/bug] Description of Bug"
labels: bug, datahub-v1.0-rc
assignees: chriscollins3456, david-leifker, maggiehays

---

**Describe the bug**
A clear and concise description of what the bug is.

**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

**Expected behavior**
A clear and concise description of what you expected to happen.

**Screenshots and/or Screen Recordings**
If applicable, add screenshots and/or screen recordings to help explain the issue.

**System details (please complete the following information):**
- DataHub Version Tag [e.g. v1.0-rc1]
- OS: [e.g. iOS]
- Browser [e.g. chrome, safari]

**Additional context**
Add any other context about the problem here.
15 changes: 10 additions & 5 deletions .github/workflows/docker-unified.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1039,12 +1039,12 @@ jobs:
cypress_matrix=$(printf "{\"test_strategy\":\"cypress\",\"batch\":\"0\",\"batch_count\":\"$cypress_batch_count\"}"; for ((i=1;i<cypress_batch_count;i++)); do printf ",{\"test_strategy\":\"cypress\", \"batch_count\":\"$cypress_batch_count\",\"batch\":\"%d\"}" $i; done)

includes=''
if [[ "${{ needs.setup.outputs.frontend_only }}" == 'true' ]]; then
includes=$cypress_matrix
elif [ "${{ needs.setup.outputs.ingestion_only }}" == 'true' ]; then
includes=$python_matrix
elif [[ "${{ needs.setup.outputs.backend_change }}" == 'true' || "${{ needs.setup.outputs.smoke_test_change }}" == 'true' ]]; then
if [[ "${{ needs.setup.outputs.backend_change }}" == 'true' || "${{ needs.setup.outputs.smoke_test_change }}" == 'true' || "${{ needs.setup.outputs.publish }}" == 'true' ]]; then
includes="$python_matrix,$cypress_matrix"
elif [[ "${{ needs.setup.outputs.frontend_only }}" == 'true' ]]; then
includes="$cypress_matrix"
elif [[ "${{ needs.setup.outputs.ingestion_only }}" == 'true' ]]; then
includes="$python_matrix"
fi
echo "matrix={\"include\":[$includes] }" >> "$GITHUB_OUTPUT"

Expand Down Expand Up @@ -1274,6 +1274,11 @@ jobs:
**/build/test-results/test/**
**/junit.*.xml
!**/binary/**
- name: Upload test results to Codecov
if: ${{ !cancelled() }}
uses: codecov/test-results-action@v1
with:
token: ${{ secrets.CODECOV_TOKEN }}
deploy_datahub_head:
name: Deploy to Datahub HEAD
runs-on: ubuntu-latest
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/metadata-model.yml
Original file line number Diff line number Diff line change
Expand Up @@ -60,4 +60,5 @@ jobs:
env:
DATAHUB_SERVER: ${{ secrets.DataHubServer }}
DATAHUB_TOKEN: ${{ secrets.DataHubToken }}
DATAHUB_TELEMETRY_ENABLED: false
run: ./gradlew :metadata-ingestion:modelDocUpload
3 changes: 2 additions & 1 deletion .github/workflows/pr-labeler.yml
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,8 @@ jobs:
"acrylJonny",
"chakru-r",
"brock-acryl",
"mminichino"
"mminichino",
"jayacryl"
]'),
github.actor
)
Expand Down
30 changes: 22 additions & 8 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Auto-generated by .github/scripts/generate_pre_commit.py at 2025-01-17 16:43:31 UTC
# Auto-generated by .github/scripts/generate_pre_commit.py at 2025-02-11 10:00:11 UTC
# Do not edit this file directly. Run the script to regenerate.
# Add additional hooks in .github/scripts/pre-commit-override.yaml
repos:
Expand Down Expand Up @@ -53,13 +53,6 @@ repos:
files: ^metadata-dao-impl/kafka-producer/.*\.java$
pass_filenames: false

- id: metadata-events-mxe-avro-spotless
name: metadata-events/mxe-avro Spotless Apply
entry: ./gradlew :metadata-events:mxe-avro:spotlessApply
language: system
files: ^metadata-events/mxe-avro/.*\.java$
pass_filenames: false

- id: metadata-events-mxe-registration-spotless
name: metadata-events/mxe-registration Spotless Apply
entry: ./gradlew :metadata-events:mxe-registration:spotlessApply
Expand Down Expand Up @@ -312,6 +305,20 @@ repos:
files: ^metadata-service/graphql-servlet-impl/.*\.java$
pass_filenames: false

- id: metadata-service-iceberg-catalog-spotless
name: metadata-service/iceberg-catalog Spotless Apply
entry: ./gradlew :metadata-service:iceberg-catalog:spotlessApply
language: system
files: ^metadata-service/iceberg-catalog/.*\.java$
pass_filenames: false

- id: metadata-service-iceberg-catalog-lint-fix
name: metadata-service/iceberg-catalog Lint Fix
entry: ./gradlew :metadata-service:iceberg-catalog:lintFix
language: system
files: ^metadata-service/iceberg-catalog/.*\.py$
pass_filenames: false

- id: metadata-service-openapi-analytics-servlet-spotless
name: metadata-service/openapi-analytics-servlet Spotless Apply
entry: ./gradlew :metadata-service:openapi-analytics-servlet:spotlessApply
Expand Down Expand Up @@ -410,6 +417,13 @@ repos:
files: ^metadata-service/servlet/.*\.java$
pass_filenames: false

- id: metadata-service-war-spotless
name: metadata-service/war Spotless Apply
entry: ./gradlew :metadata-service:war:spotlessApply
language: system
files: ^metadata-service/war/.*\.java$
pass_filenames: false

- id: metadata-utils-spotless
name: metadata-utils Spotless Apply
entry: ./gradlew :metadata-utils:spotlessApply
Expand Down
7 changes: 5 additions & 2 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ buildscript {
ext.springVersion = '6.1.14'
ext.springBootVersion = '3.2.9'
ext.springKafkaVersion = '3.1.6'
ext.openTelemetryVersion = '1.18.0'
ext.openTelemetryVersion = '1.45.0'
ext.neo4jVersion = '5.20.0'
ext.neo4jTestVersion = '5.20.0'
ext.neo4jApocVersion = '5.20.0'
Expand Down Expand Up @@ -220,7 +220,10 @@ project.ext.externalDependency = [
'neo4jApocCore': 'org.neo4j.procedure:apoc-core:' + neo4jApocVersion,
'neo4jApocCommon': 'org.neo4j.procedure:apoc-common:' + neo4jApocVersion,
'opentelemetryApi': 'io.opentelemetry:opentelemetry-api:' + openTelemetryVersion,
'opentelemetryAnnotations': 'io.opentelemetry:opentelemetry-extension-annotations:' + openTelemetryVersion,
'opentelemetrySdk': 'io.opentelemetry:opentelemetry-sdk:' + openTelemetryVersion,
'opentelemetrySdkTrace': 'io.opentelemetry:opentelemetry-sdk-trace:' + openTelemetryVersion,
'opentelemetryAutoConfig': 'io.opentelemetry:opentelemetry-sdk-extension-autoconfigure:' + openTelemetryVersion,
'opentelemetryAnnotations': 'io.opentelemetry.instrumentation:opentelemetry-instrumentation-annotations:2.11.0',
'opentracingJdbc':'io.opentracing.contrib:opentracing-jdbc:0.2.15',
'parquet': 'org.apache.parquet:parquet-avro:1.12.3',
'parquetHadoop': 'org.apache.parquet:parquet-hadoop:1.13.1',
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ private Constants() {}
public static final String CONTRACTS_SCHEMA_FILE = "contract.graphql";
public static final String CONNECTIONS_SCHEMA_FILE = "connection.graphql";
public static final String VERSION_SCHEMA_FILE = "versioning.graphql";
public static final String QUERY_SCHEMA_FILE = "query.graphql";
public static final String BROWSE_PATH_DELIMITER = "/";
public static final String BROWSE_PATH_V2_DELIMITER = "␟";
public static final String VERSION_STAMP_FIELD_NAME = "versionStamp";
Expand Down
Loading