Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DM-44092: Remove placeholder timeseries feature columns from DIAObject schemas #218

Merged
merged 1 commit into from
May 7, 2024
Merged

DM-44092: Remove placeholder timeseries feature columns from DIAObject schemas #218

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 1 addition & 85 deletions yml/apdb.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
name: "ApdbSchema"
"@id": "#apdbSchema"
version: "1.0.0"
version: "1.1.0"
Copy link
Member

@kfindeisen kfindeisen May 7, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since it seems like ApdbSchema is using a "fully/partially/non-compatible" approach instead of semantic versioning, I don't think this number is correct -- if you want to argue that it's compatible because the fields were unused, then it should be 1.0.1; if you want to consider the schema on its own merits, it's 2.0.0. I assume the former is closer to your original intent.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can see arguments for either, but @andy-slac had suggested 1.1.0 in a DM so I went with that.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is a complicated issue because it also involves something on client side. Semantic versioning is also hard to apply to database schema (semantic versioning as mostly about API changes). In this case my thinking was that we cannot claim full compatibility because there may be a potential client that will read 1.0.0 schema and try to do something with those columns. Backward compatibility seems safer - client that is configured with 1.1.0 schema can still read and write 1.0.0 database.

Copy link
Member

@kfindeisen kfindeisen May 7, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit worried by the definition of "backward compatibility" implied by that statement.

Normally when people talk about a library (or its API changes) being backward compatible, they mean that old clients can use new versions of the library. That's also what Prompt Processing assumed it meant.

However, you seem to be saying that an APDB change is compatible if new clients can interact with old databases, i.e. forward compatibility in the conventional sense. That's surprising because such compatibility is normally the client's responsibility. It would also mean that in PP we need to be stricter about saying that e.g. a given release can work with schemas 1.0-1.1, but can't guarantee 1.2.

If that's the direction you want compatibility to be defined in, please document that very explicitly.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, maybe term "backward" is not quite right here, but it more or less reflects the same direction of upgrades. With API example you upgrade library and keep old client. With schema version - you upgrade yaml schema file but can keep old database schema. I'm not sure one can say who is the client of what here, but upgrades usually happen for yaml schema first.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still, please make that unambiguous in the docs (in practice, DMTN-269?) so that everybody has the same expectations.

description: The Alert Production Database (APDB) contains the catalogs resulting from
image differencing during nightly Prompt Processing as well as the results of
daily Solar System Processing.
Expand Down Expand Up @@ -424,90 +424,6 @@ tables:
description: Standard deviation of the distribution of y_fpFlux.
fits:tunit: nJy
ivoa:ucd: stat.stdev
- name: u_lcPeriodic
"@id": "#DiaObject.u_lcPeriodic"
datatype: binary
length: 128
description: Periodic features extracted from light-curves using generalized Lomb-Scargle
periodogram for u filter. [32 FLOAT].
mysql:datatype: BLOB
- name: g_lcPeriodic
"@id": "#DiaObject.g_lcPeriodic"
datatype: binary
length: 128
description: Periodic features extracted from light-curves using generalized Lomb-Scargle
periodogram for g filter. [32 FLOAT].
mysql:datatype: BLOB
- name: r_lcPeriodic
"@id": "#DiaObject.r_lcPeriodic"
datatype: binary
length: 128
description: Periodic features extracted from light-curves using generalized Lomb-Scargle
periodogram for r filter. [32 FLOAT].
mysql:datatype: BLOB
- name: i_lcPeriodic
"@id": "#DiaObject.i_lcPeriodic"
datatype: binary
length: 128
description: Periodic features extracted from light-curves using generalized Lomb-Scargle
periodogram for i filter. [32 FLOAT].
mysql:datatype: BLOB
- name: z_lcPeriodic
"@id": "#DiaObject.z_lcPeriodic"
datatype: binary
length: 128
description: Periodic features extracted from light-curves using generalized Lomb-Scargle
periodogram for z filter. [32 FLOAT].
mysql:datatype: BLOB
- name: y_lcPeriodic
"@id": "#DiaObject.y_lcPeriodic"
datatype: binary
length: 128
description: Periodic features extracted from light-curves using generalized Lomb-Scargle
periodogram for y filter. [32 FLOAT].
mysql:datatype: BLOB
- name: u_lcNonPeriodic
"@id": "#DiaObject.u_lcNonPeriodic"
datatype: binary
length: 80
description: Non-periodic features extracted from light-curves using generalized
Lomb-Scargle periodogram for u filter. [20 FLOAT].
mysql:datatype: BLOB
- name: g_lcNonPeriodic
"@id": "#DiaObject.g_lcNonPeriodic"
datatype: binary
length: 80
description: Non-periodic features extracted from light-curves using generalized
Lomb-Scargle periodogram for g filter. [20 FLOAT].
mysql:datatype: BLOB
- name: r_lcNonPeriodic
"@id": "#DiaObject.r_lcNonPeriodic"
datatype: binary
length: 80
description: Non-periodic features extracted from light-curves using generalized
Lomb-Scargle periodogram for r filter. [20 FLOAT].
mysql:datatype: BLOB
- name: i_lcNonPeriodic
"@id": "#DiaObject.i_lcNonPeriodic"
datatype: binary
length: 80
description: Non-periodic features extracted from light-curves using generalized
Lomb-Scargle periodogram for i filter. [20 FLOAT].
mysql:datatype: BLOB
- name: z_lcNonPeriodic
"@id": "#DiaObject.z_lcNonPeriodic"
datatype: binary
length: 80
description: Non-periodic features extracted from light-curves using generalized
Lomb-Scargle periodogram for z filter. [20 FLOAT].
mysql:datatype: BLOB
- name: y_lcNonPeriodic
"@id": "#DiaObject.y_lcNonPeriodic"
datatype: binary
length: 80
description: Non-periodic features extracted from light-curves using generalized
Lomb-Scargle periodogram for y filter. [20 FLOAT].
mysql:datatype: BLOB
- name: nearbyObj1
"@id": "#DiaObject.nearbyObj1"
datatype: long
Expand Down
Loading