Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dotCMS CLI: Implement files push command #24740

Closed
6 tasks
nollymar opened this issue Apr 24, 2023 · 1 comment · Fixed by #25714
Closed
6 tasks

dotCMS CLI: Implement files push command #24740

nollymar opened this issue Apr 24, 2023 · 1 comment · Fixed by #25714

Comments

@nollymar
Copy link
Contributor

nollymar commented Apr 24, 2023

Parent Issue

#24732

User Story

As a user, I'd like to execute the files push command to upload file(s) or/and folder(s) from my local machine to a remote server (dotCMS) by specifying the location of the local directory or file.

Examples:

files push /cli/project/workspace/path --dry-run
files push /cli/project/workspace/path
files push /cli/project/workspace/path/files/live/es-es/demo.dotcms.com
files push /cli/project/workspace/path/files/live/en-us/demo.dotcms.com/images
files push /cli/project/workspace/path/files/live/en-us/demo.dotcms.com/images --dry-run
files push /cli/project/workspace/path/files/live/en-us/demo.dotcms.com/application/themes/landing-page/header.vtl

This command should consume the endpoint implemented here: #24738

Acceptance Criteria

  • The "push" operation should allow the user to upload file(s) or/and folder(s) from their local machine to dotCMS.
  • The user should be able to preview changes that would be made on the remote server without applying them through the "--dry-run" option.
  • The "push" operation should display appropriate progress messages as the file(s) or/and folder(s) are uploaded.
  • If a file is archived, we should unarchive/publish it
  • Make sure zero length files can be pushed. Related to Error when update zero size file through Webdav or CLI. #24042
  • Implement tests accordingly

Proposed Objective

Core Features

Proposed Priority

Priority 3 - Average

@nollymar nollymar moved this from New to Current Sprint Backlog in dotCMS - Product Planning Apr 24, 2023
@jgambarios jgambarios moved this from Current Sprint Backlog to In Progress in dotCMS - Product Planning Jun 27, 2023
@jgambarios jgambarios self-assigned this Jun 27, 2023
jgambarios added a commit that referenced this issue Jul 7, 2023
jgambarios added a commit that referenced this issue Jul 10, 2023
jgambarios added a commit that referenced this issue Jul 11, 2023
jgambarios added a commit that referenced this issue Jul 18, 2023
jgambarios added a commit that referenced this issue Jul 18, 2023
jgambarios added a commit that referenced this issue Jul 26, 2023
jgambarios added a commit that referenced this issue Jul 26, 2023
jgambarios added a commit that referenced this issue Jul 26, 2023
jgambarios added a commit that referenced this issue Jul 29, 2023
Added a 'fail-fast' option to the FilesPush and FilesPull commands, which enables the command to stop immediately after encountering the first error during the push or pull process. Updated error handling to log all errors encountered during the push or pull process, instead of only logging the first error.
jgambarios added a commit that referenced this issue Jul 29, 2023
Code has been modified to utilize the File class for managing the destination directory in the file pull and push operations. Previously, the destination directory was managed with a combination of String and Path objects. This caused unnecessary complexity and potential for errors, so the refactor simplifies the codebase to only use the File object instead. Additionally, unnecessary methods and imports related to the old implementation have been removed. The workspace-related functionality is moved to `AbstractFilesCommand`. This refactor improves overall code readability, maintenance, and possibly performance.
jgambarios added a commit that referenced this issue Jul 31, 2023
The LOCATION_FILES constant was replaced with Workspace.FILES_NAMESPACE for consistency throughout the code.
Adding a SiteCreationException class to handle site creation errors specifically.
The error messages returned by the system are now more explicit and provide the related exception message,
assisting in troubleshooting. Also, if a site creation fails, the system will now stop the execution immediately.
Several usability improvements are also introduced, such as the improved formatting of error output and handling of the workspace directory.
jgambarios added a commit that referenced this issue Jul 31, 2023
Change the method name from `buildFileSystemTree` to `pullTreeNode` and modified the corresponding task's name and description to `PullTreeNodeTask` from `LocalFoldersTreeBuilderTask`. The change is made to more accurately reflect the method/task's functionality which is to pull the contents of a tree node from a remote server rather than just building a local file system tree.
jgambarios added a commit that referenced this issue Aug 1, 2023
Implemented the ability to specify a maximum number of retry attempts for the push and pull content processes. This allows the tasks to recover from temporary errors by attempting to retry the operation a specified number of times before failing completely.
This feature was added by introducing 'maxRetryAttempts' as a parameter to the pullFile, pullTree, and processTreeNodes methods and implementing retry logic in PushTreeNodeTask, PullTreeNodeTask, PullTreeNodeTask, and PullFile classes. Retries occur when exceptions are thrown during the push or pull processes. These can be particularly useful in network environments where temporary connectivity issues can occur. The retry count starts at 0 and after each failure, we increment it by one until it reaches the maxRetryAttempts count.
jgambarios added a commit that referenced this issue Aug 3, 2023
Renamed the test class from `FolderTraversalServiceTest` to `RemoteTraversalServiceTest`. Implemented new checks for folder existence and file count within specific folders. Also added test cases for inclusion and exclusion of specific file types. These changes provide a better scope in testing the traversal of files within folders remotely.
jgambarios added a commit that referenced this issue Aug 4, 2023
Implemented various tests for the file push service, `PushServiceTest`, that checks different scenarios such as pushing existing content, pushing new content, and pushing modifications. Refactored the existing `PullServiceTest` to test the functionality in a more comprehensive way, including file and folder presence checks after pulling content, and considers situations of included and excluded file types. These tests ensure both services perform as expected in various usage scenarios.
jgambarios added a commit that referenced this issue Aug 4, 2023
Deleted AssetsUtilsTest.java and created LocationUtilsTest.java, both under /api-data-model/src/test/java/com/dotcms/common. Referencing the URL folder check was moved from AssetsUtils to LocationUtils, hence these tests have also been moved. LocationUtils class better aligns with the testing intent as it is fundamentally dealing with URL and directory structures. This change should make our test setup more intuitive and maintainable.
jgambarios added a commit that referenced this issue Aug 7, 2023
Renamed several methods in AssetsUtils and LocationUtils classes adhering to standard naming conventions. The "isFolder" and "localPathFromAssetData" methods were renamed to "isFolderURL", "isFolderURI" and "localPathFromAssetData" respectively, to provide better clarity. The change has been reflected in FilesPull and Pusher classes as well. Additionally, updated the LocationUtilsTest class with the renamed methods to ensure test cases are aligned with these changes and maintain code integrity.
jgambarios added a commit that referenced this issue Aug 7, 2023
Multiple methods in the AssetsUtils and FilesUtils classes were renamed to follow standard Java naming conventions. The change improves code readability and clarity. All occurrences in the code where these methods were used have been updated and the relevant test cases have been adjusted accordingly to reflect these changes.
nollymar pushed a commit that referenced this issue Aug 7, 2023
* #24740 Working on the push command

* #24740 Refactoring

* #24740

* #24740 Better handling of empty folders

* #24740 Working on second step, pushing the content to the remote server

* #24740 Working on second step, pushing the content to the remote server

* #24740 Adding options for dry-run, and remove assets and folders.

* #24740 Adding options for dry-run, and remove assets and folders.

* #24740

* #24740

* #24740 Fixing creation of languages when does not exist

* #24740 Fixing a case for the glob filter and assets in the root

* #24740

* #24740

* #24740

* #24740 Update pushService parameters in FilesPush

* #24740 Update fail-fast and error handling during file pull and push.

Added a 'fail-fast' option to the FilesPush and FilesPull commands, which enables the command to stop immediately after encountering the first error during the push or pull process. Updated error handling to log all errors encountered during the push or pull process, instead of only logging the first error.

* #24740

Code has been modified to utilize the File class for managing the destination directory in the file pull and push operations. Previously, the destination directory was managed with a combination of String and Path objects. This caused unnecessary complexity and potential for errors, so the refactor simplifies the codebase to only use the File object instead. Additionally, unnecessary methods and imports related to the old implementation have been removed. The workspace-related functionality is moved to `AbstractFilesCommand`. This refactor improves overall code readability, maintenance, and possibly performance.

* #24740 Refactor processTreeNodes method to handle error logging

* #24740 Refactor location files namespace and improve error handling

The LOCATION_FILES constant was replaced with Workspace.FILES_NAMESPACE for consistency throughout the code.
Adding a SiteCreationException class to handle site creation errors specifically.
The error messages returned by the system are now more explicit and provide the related exception message,
assisting in troubleshooting. Also, if a site creation fails, the system will now stop the execution immediately.
Several usability improvements are also introduced, such as the improved formatting of error output and handling of the workspace directory.

* #24740 Rename file and method names for clarity

Change the method name from `buildFileSystemTree` to `pullTreeNode` and modified the corresponding task's name and description to `PullTreeNodeTask` from `LocalFoldersTreeBuilderTask`. The change is made to more accurately reflect the method/task's functionality which is to pull the contents of a tree node from a remote server rather than just building a local file system tree.

* #24740 Add retry attempts to push and pull content processes

Implemented the ability to specify a maximum number of retry attempts for the push and pull content processes. This allows the tasks to recover from temporary errors by attempting to retry the operation a specified number of times before failing completely.
This feature was added by introducing 'maxRetryAttempts' as a parameter to the pullFile, pullTree, and processTreeNodes methods and implementing retry logic in PushTreeNodeTask, PullTreeNodeTask, PullTreeNodeTask, and PullFile classes. Retries occur when exceptions are thrown during the push or pull processes. These can be particularly useful in network environments where temporary connectivity issues can occur. The retry count starts at 0 and after each failure, we increment it by one until it reaches the maxRetryAttempts count.

* #24740 Improve error handling in file traversal tasks

Introduced a custom exception, TraversalTaskException, to enhance error handling during the execution of traversal tasks in file operations. This exception will provide more context about errors occurring during the traversal process. The changes include:
- Replacing generic RuntimeException with the new TraversalTaskException in various methods across the service layer that perform file operations.
- Updating tests and adding new ones to verify the correct functionality of updated methods.
- Adding and importing necessary dependencies in pom.xml to support the changes.
This change is essential to ensure a better debugging experience by eliminating unnecessary noise from error logs and providing more precise error messages.

* #24740 Rename class and extend folder traversal testing

Renamed the test class from `FolderTraversalServiceTest` to `RemoteTraversalServiceTest`. Implemented new checks for folder existence and file count within specific folders. Also added test cases for inclusion and exclusion of specific file types. These changes provide a better scope in testing the traversal of files within folders remotely.

* #24740 Add tests for file push service and refactor pull service test

Implemented various tests for the file push service, `PushServiceTest`, that checks different scenarios such as pushing existing content, pushing new content, and pushing modifications. Refactored the existing `PullServiceTest` to test the functionality in a more comprehensive way, including file and folder presence checks after pulling content, and considers situations of included and excluded file types. These tests ensure both services perform as expected in various usage scenarios.

* #24740 Refactor AssetUtils tests to LocationUtils

Deleted AssetsUtilsTest.java and created LocationUtilsTest.java, both under /api-data-model/src/test/java/com/dotcms/common. Referencing the URL folder check was moved from AssetsUtils to LocationUtils, hence these tests have also been moved. LocationUtils class better aligns with the testing intent as it is fundamentally dealing with URL and directory structures. This change should make our test setup more intuitive and maintainable.

* #24740 Refactor method naming for improved clarity

Renamed several methods in AssetsUtils and LocationUtils classes adhering to standard naming conventions. The "isFolder" and "localPathFromAssetData" methods were renamed to "isFolderURL", "isFolderURI" and "localPathFromAssetData" respectively, to provide better clarity. The change has been reflected in FilesPull and Pusher classes as well. Additionally, updated the LocationUtilsTest class with the renamed methods to ensure test cases are aligned with these changes and maintain code integrity.

* Add docstrings to FilterTest and refactor imports in multiple files

This commit includes adding descriptive docstrings to test methods in FilterTest.java to document their intended functionality. It also includes minor refactoring of import statements in FilterTest.java, RemoteTraversalServiceTest.java, PushServiceTest.java, and PullServiceTest.java. The refactoring simplifies code structure for improved readability without changing the functionality of the code. The commit aims to enhance understanding of the test cases and to adhere to best practices for code organization.

* #24740 Refactor method naming to adhere to Java naming conventions

Multiple methods in the AssetsUtils and FilesUtils classes were renamed to follow standard Java naming conventions. The change improves code readability and clarity. All occurrences in the code where these methods were used have been updated and the relevant test cases have been adjusted accordingly to reflect these changes.
@nollymar nollymar linked a pull request Aug 7, 2023 that will close this issue
@nollymar nollymar moved this from In Progress to Internal QA in dotCMS - Product Planning Aug 7, 2023
@dcolina dcolina self-assigned this Aug 9, 2023
@dcolina
Copy link
Contributor

dcolina commented Aug 9, 2023

Note: I have noticed that some actions have changed in definition, so it would be good to update the user history to avoid confusion.

This can be checked by looking at the result of the. files push --help command

Otherwise, the task meets each of the requirements specified in the acceptance criteria.

Screenshot 2023-08-09 at 16 05 57

Test cases:

Push a new file

Given:

A new image file on working status and es-es lang, in this case we add kunisports-1.jpg.

Image

When:

Run files push command.

Image

Then you can see the new file pushed and in draft status.

![Image](https://github.com/dotCMS/core/assets/29883881/bd044c59-1ced-441d-a473-037b656d758b

Push file with dry-run option

Given:

A new folder and a new file in it on live status and en-us lang, in this case we add nested-folder/jijantes-fc.svg

Image

When:

Run files push --dry-run command.

Image

Then:

You can see the new folder and the new file in the projection.

Push an empty folder

Given:

A new folder (empty) in it on live status and en-us lang, in this case we add nested-folder/

Image

When:

Run files push command.

Image

Remove a folder and its content

Given:

Remove the event folder and its content on live status and en-us lang, in this case events

Image

When:

Run files push --removeAssets command.

Image

Push with empty-file (zero bytes)

Given:

A new empty file (zero bytes).

Image

When:

Run files push

Image

Then:

Image

Push with empty-folder

Given:

A new empty folder.

Image

When:

Run files push

Image

Then:

The empty folder is ignored.

Image

Remove folders and assets

Given:

Delete an existing folder and its all content.

Image

When:

Run files push --removeAssets --removeFolders command.

Image

Then:

You can see the folder new-folder and its content have been disappeared.

Image

Fail fast option

Given:

  • Add a image file. In this case we added kunisports-1.jpg
  • Add some files to force errors. In this case we added unsupported files like .json or .svg files.

Screenshot 2023-08-10 at 18 31 15

When:

Run files push --fail-fast command.

Then:

Screenshot 2023-08-10 at 18 32 12

Now we removed the file with errors.

Screenshot 2023-08-10 at 18 34 21

When:

Run files push --fail-fast command again.

Then:

Screenshot 2023-08-10 at 18 35 58

Successfully execution, this means the command is working fine.

Internal QA: PASSED.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

4 participants