Skip to content
This repository has been archived by the owner on Oct 11, 2024. It is now read-only.

merge backup details from base snapshot(s) and items in Collections #1800

Closed
Tracked by #1777
ashmrtn opened this issue Dec 13, 2022 · 0 comments · Fixed by #1905
Closed
Tracked by #1777

merge backup details from base snapshot(s) and items in Collections #1800

ashmrtn opened this issue Dec 13, 2022 · 0 comments · Fixed by #1905
Assignees
@ashmrtn
Labels
enhancement New feature or request

Comments

@ashmrtn
Copy link
Contributor

ashmrtn commented Dec 13, 2022 •
edited
Loading

Incremental backups should still have a complete set of backup details with an entry for each item even if the item was not reported as changed/added/deleted by GraphConnector

Backup details sometimes contains information that is only available in the data stored in kopia files. This means that for unchanged items that are not reported by GraphConnector we need a way to source this information. Using the backup details of the base snapshot(s) is a good starting point as it already contains the info we need

Items that are added/updated and reported by GraphConnector are assumed to have the correct backup details
@ashmrtn ashmrtn mentioned this issue Dec 13, 2022
Closed
29 tasks
This was referenced Dec 14, 2022
Closed
Merged
aviator-app bot pushed a commit that referenced this issue Dec 15, 2022
@vkamra
Add an updated flag to backup details (#1813)
debe74a 
## Description

Adds a flag in backup details that indicates whether the item in that backup is new/updated.

Currently always set to `true` but once we implement #1800 - we will set this to false for existing items from
base snapshots.

## Does this PR need a docs update or release note?

- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #1812 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
@ashmrtn ashmrtn self-assigned this Dec 19, 2022
@ashmrtn ashmrtn added the enhancement New feature or request label Dec 19, 2022
This was referenced Dec 20, 2022
Merged
Merged
Merged
Merged
Merged
Merged
aviator-app bot pushed a commit that referenced this issue Dec 21, 2022
@ashmrtn
Move OneDrive path helper func to path package (#1880)
df086bd 
## Description

Later code in details package will need access to this function so it can update item details. However, importing the onedrive package in the details package leads to an import cycle. Moving it to the path package breaks the cycle while still allowing the code to be accessed.

## Does this PR need a docs update or release note?

- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

* #1800 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
aviator-app bot pushed a commit that referenced this issue Dec 22, 2022
@ashmrtn
Allow updating portions of ItemInfo for some types (#1883)
d3573d1 
## Description

Add helper functions that allows updating ItemInfo given a new path.Path. This is required because some items embed portions of the path in their details.

## Does this PR need a docs update or release note?

- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No 

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #1800 

## Test Plan

- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
aviator-app bot pushed a commit that referenced this issue Dec 22, 2022
@ashmrtn
Factor out function to retrieve backup details (#1894)
f37ee2b 
## Description

Given a backup ID pull the backup details for the backup. This function will also be used during incremental backups as the backup details from the base snapshot(s) will need merged with the details generated by backing up new/changed items.

## Does this PR need a docs update or release note?

- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No 

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

* #1800 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [x] 💚 E2E
aviator-app bot pushed a commit that referenced this issue Dec 22, 2022
@ashmrtn
Produce map of items that need backup details entries (#1892)
6b689b7 
## Description

Generate a `map[old ShortRef]new path.Path` for every item sourced from a base snapshot during backup. Return this information at the end of BackupCollections so that callers can use it to merge backup details if desired.

This PR does a few auxiliary things as well:
* thread current and previous path through hierarchy merging
* add extra field to info tracked by corsoProgress to hold previous path
* provide current and previous paths to streamBaseItems function

## Does this PR need a docs update or release note?

- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No 

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #1800 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
@ashmrtn ashmrtn mentioned this issue Dec 22, 2022
Merged
12 tasks
aviator-app bot pushed a commit that referenced this issue Dec 22, 2022
@ashmrtn
Lift'n'shift of existing code (#1925)
04851f4 
## Description

Factor out code to get a set of FolderEntries based on some path information. This code will be used in BackupOp when merging item details.

## Does this PR need a docs update or release note?

- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No 

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

* #1800 

## Test Plan

- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
@ashmrtn ashmrtn mentioned this issue Dec 23, 2022
Merged
12 tasks
aviator-app bot pushed a commit that referenced this issue Dec 23, 2022
@ashmrtn
Functions to merge a set of backup details (#1904)
bc56d38 
## Description

Go through the provided bases, load their backup details, and check if
any of the items in them need to be merged into the details for the
current backup.

Has a small amount of logic to treat moved items as updated.

Also include all the tests

Viewing by commit may be useful

## Does this PR need a docs update or release note?

- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No 

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #1800 

## Test Plan

- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
aviator-app bot pushed a commit that referenced this issue Dec 23, 2022
@ashmrtn
Update details merge function for details Builder and new location of…
62c8f8c 
… folder population (#1926)

## Description

Folder population now done when merging items in BackupOp. Also add more tests to make sure folders are actually populated properly in Details

## Does this PR need a docs update or release note?

- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No 

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #1800 

## Test Plan

- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
aviator-app bot pushed a commit that referenced this issue Dec 23, 2022
@ashmrtn
Wire details merging during BackupOp.Run (#1905)
35c1b10 
## Description

Actually call the details merge function while performing a backup. If no base snapshots or `len(toMerge) == 0` then turns into a noop.

This PR contains a little extra refactoring to keep from initializing a struct multiple times. However, long-term we should probably refactor BackupOp to better fit the fact that we are now using interfaces as inputs to many functions for the sake of testing

## Does this PR need a docs update or release note?

- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No 

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #1800 

## Test Plan

- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@ashmrtn ashmrtn

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Wire details merging during BackupOp.Run alcionai/corso
1 participant
@ashmrtn