-
Notifications
You must be signed in to change notification settings - Fork 470
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
dotCMS CLI: Implement files push command #24740
Comments
Added a 'fail-fast' option to the FilesPush and FilesPull commands, which enables the command to stop immediately after encountering the first error during the push or pull process. Updated error handling to log all errors encountered during the push or pull process, instead of only logging the first error.
Code has been modified to utilize the File class for managing the destination directory in the file pull and push operations. Previously, the destination directory was managed with a combination of String and Path objects. This caused unnecessary complexity and potential for errors, so the refactor simplifies the codebase to only use the File object instead. Additionally, unnecessary methods and imports related to the old implementation have been removed. The workspace-related functionality is moved to `AbstractFilesCommand`. This refactor improves overall code readability, maintenance, and possibly performance.
The LOCATION_FILES constant was replaced with Workspace.FILES_NAMESPACE for consistency throughout the code. Adding a SiteCreationException class to handle site creation errors specifically. The error messages returned by the system are now more explicit and provide the related exception message, assisting in troubleshooting. Also, if a site creation fails, the system will now stop the execution immediately. Several usability improvements are also introduced, such as the improved formatting of error output and handling of the workspace directory.
Change the method name from `buildFileSystemTree` to `pullTreeNode` and modified the corresponding task's name and description to `PullTreeNodeTask` from `LocalFoldersTreeBuilderTask`. The change is made to more accurately reflect the method/task's functionality which is to pull the contents of a tree node from a remote server rather than just building a local file system tree.
Implemented the ability to specify a maximum number of retry attempts for the push and pull content processes. This allows the tasks to recover from temporary errors by attempting to retry the operation a specified number of times before failing completely. This feature was added by introducing 'maxRetryAttempts' as a parameter to the pullFile, pullTree, and processTreeNodes methods and implementing retry logic in PushTreeNodeTask, PullTreeNodeTask, PullTreeNodeTask, and PullFile classes. Retries occur when exceptions are thrown during the push or pull processes. These can be particularly useful in network environments where temporary connectivity issues can occur. The retry count starts at 0 and after each failure, we increment it by one until it reaches the maxRetryAttempts count.
Renamed the test class from `FolderTraversalServiceTest` to `RemoteTraversalServiceTest`. Implemented new checks for folder existence and file count within specific folders. Also added test cases for inclusion and exclusion of specific file types. These changes provide a better scope in testing the traversal of files within folders remotely.
Implemented various tests for the file push service, `PushServiceTest`, that checks different scenarios such as pushing existing content, pushing new content, and pushing modifications. Refactored the existing `PullServiceTest` to test the functionality in a more comprehensive way, including file and folder presence checks after pulling content, and considers situations of included and excluded file types. These tests ensure both services perform as expected in various usage scenarios.
Deleted AssetsUtilsTest.java and created LocationUtilsTest.java, both under /api-data-model/src/test/java/com/dotcms/common. Referencing the URL folder check was moved from AssetsUtils to LocationUtils, hence these tests have also been moved. LocationUtils class better aligns with the testing intent as it is fundamentally dealing with URL and directory structures. This change should make our test setup more intuitive and maintainable.
Renamed several methods in AssetsUtils and LocationUtils classes adhering to standard naming conventions. The "isFolder" and "localPathFromAssetData" methods were renamed to "isFolderURL", "isFolderURI" and "localPathFromAssetData" respectively, to provide better clarity. The change has been reflected in FilesPull and Pusher classes as well. Additionally, updated the LocationUtilsTest class with the renamed methods to ensure test cases are aligned with these changes and maintain code integrity.
Multiple methods in the AssetsUtils and FilesUtils classes were renamed to follow standard Java naming conventions. The change improves code readability and clarity. All occurrences in the code where these methods were used have been updated and the relevant test cases have been adjusted accordingly to reflect these changes.
* #24740 Working on the push command * #24740 Refactoring * #24740 * #24740 Better handling of empty folders * #24740 Working on second step, pushing the content to the remote server * #24740 Working on second step, pushing the content to the remote server * #24740 Adding options for dry-run, and remove assets and folders. * #24740 Adding options for dry-run, and remove assets and folders. * #24740 * #24740 * #24740 Fixing creation of languages when does not exist * #24740 Fixing a case for the glob filter and assets in the root * #24740 * #24740 * #24740 * #24740 Update pushService parameters in FilesPush * #24740 Update fail-fast and error handling during file pull and push. Added a 'fail-fast' option to the FilesPush and FilesPull commands, which enables the command to stop immediately after encountering the first error during the push or pull process. Updated error handling to log all errors encountered during the push or pull process, instead of only logging the first error. * #24740 Code has been modified to utilize the File class for managing the destination directory in the file pull and push operations. Previously, the destination directory was managed with a combination of String and Path objects. This caused unnecessary complexity and potential for errors, so the refactor simplifies the codebase to only use the File object instead. Additionally, unnecessary methods and imports related to the old implementation have been removed. The workspace-related functionality is moved to `AbstractFilesCommand`. This refactor improves overall code readability, maintenance, and possibly performance. * #24740 Refactor processTreeNodes method to handle error logging * #24740 Refactor location files namespace and improve error handling The LOCATION_FILES constant was replaced with Workspace.FILES_NAMESPACE for consistency throughout the code. Adding a SiteCreationException class to handle site creation errors specifically. The error messages returned by the system are now more explicit and provide the related exception message, assisting in troubleshooting. Also, if a site creation fails, the system will now stop the execution immediately. Several usability improvements are also introduced, such as the improved formatting of error output and handling of the workspace directory. * #24740 Rename file and method names for clarity Change the method name from `buildFileSystemTree` to `pullTreeNode` and modified the corresponding task's name and description to `PullTreeNodeTask` from `LocalFoldersTreeBuilderTask`. The change is made to more accurately reflect the method/task's functionality which is to pull the contents of a tree node from a remote server rather than just building a local file system tree. * #24740 Add retry attempts to push and pull content processes Implemented the ability to specify a maximum number of retry attempts for the push and pull content processes. This allows the tasks to recover from temporary errors by attempting to retry the operation a specified number of times before failing completely. This feature was added by introducing 'maxRetryAttempts' as a parameter to the pullFile, pullTree, and processTreeNodes methods and implementing retry logic in PushTreeNodeTask, PullTreeNodeTask, PullTreeNodeTask, and PullFile classes. Retries occur when exceptions are thrown during the push or pull processes. These can be particularly useful in network environments where temporary connectivity issues can occur. The retry count starts at 0 and after each failure, we increment it by one until it reaches the maxRetryAttempts count. * #24740 Improve error handling in file traversal tasks Introduced a custom exception, TraversalTaskException, to enhance error handling during the execution of traversal tasks in file operations. This exception will provide more context about errors occurring during the traversal process. The changes include: - Replacing generic RuntimeException with the new TraversalTaskException in various methods across the service layer that perform file operations. - Updating tests and adding new ones to verify the correct functionality of updated methods. - Adding and importing necessary dependencies in pom.xml to support the changes. This change is essential to ensure a better debugging experience by eliminating unnecessary noise from error logs and providing more precise error messages. * #24740 Rename class and extend folder traversal testing Renamed the test class from `FolderTraversalServiceTest` to `RemoteTraversalServiceTest`. Implemented new checks for folder existence and file count within specific folders. Also added test cases for inclusion and exclusion of specific file types. These changes provide a better scope in testing the traversal of files within folders remotely. * #24740 Add tests for file push service and refactor pull service test Implemented various tests for the file push service, `PushServiceTest`, that checks different scenarios such as pushing existing content, pushing new content, and pushing modifications. Refactored the existing `PullServiceTest` to test the functionality in a more comprehensive way, including file and folder presence checks after pulling content, and considers situations of included and excluded file types. These tests ensure both services perform as expected in various usage scenarios. * #24740 Refactor AssetUtils tests to LocationUtils Deleted AssetsUtilsTest.java and created LocationUtilsTest.java, both under /api-data-model/src/test/java/com/dotcms/common. Referencing the URL folder check was moved from AssetsUtils to LocationUtils, hence these tests have also been moved. LocationUtils class better aligns with the testing intent as it is fundamentally dealing with URL and directory structures. This change should make our test setup more intuitive and maintainable. * #24740 Refactor method naming for improved clarity Renamed several methods in AssetsUtils and LocationUtils classes adhering to standard naming conventions. The "isFolder" and "localPathFromAssetData" methods were renamed to "isFolderURL", "isFolderURI" and "localPathFromAssetData" respectively, to provide better clarity. The change has been reflected in FilesPull and Pusher classes as well. Additionally, updated the LocationUtilsTest class with the renamed methods to ensure test cases are aligned with these changes and maintain code integrity. * Add docstrings to FilterTest and refactor imports in multiple files This commit includes adding descriptive docstrings to test methods in FilterTest.java to document their intended functionality. It also includes minor refactoring of import statements in FilterTest.java, RemoteTraversalServiceTest.java, PushServiceTest.java, and PullServiceTest.java. The refactoring simplifies code structure for improved readability without changing the functionality of the code. The commit aims to enhance understanding of the test cases and to adhere to best practices for code organization. * #24740 Refactor method naming to adhere to Java naming conventions Multiple methods in the AssetsUtils and FilesUtils classes were renamed to follow standard Java naming conventions. The change improves code readability and clarity. All occurrences in the code where these methods were used have been updated and the relevant test cases have been adjusted accordingly to reflect these changes.
Note: I have noticed that some actions have changed in definition, so it would be good to update the user history to avoid confusion. This can be checked by looking at the result of the. Otherwise, the task meets each of the requirements specified in the acceptance criteria. Test cases: Push a new fileGiven: A new image file on When: Run Then you can see the new file pushed and in draft status.  in it on When: Run Remove a folder and its contentGiven: Remove the event folder and its content on When: Run Push with empty-file (zero bytes)Given: A new empty file (zero bytes). When: Run Then: Push with empty-folderGiven: A new empty folder. When: Run Then: The empty folder is ignored. Remove folders and assetsGiven: Delete an existing folder and its all content. When: Run Then: You can see the folder Fail fast optionGiven:
When: Run Then: Now we removed the file with errors. When: Run Then: Successfully execution, this means the command is working fine. Internal QA: PASSED. |
Parent Issue
#24732
User Story
As a user, I'd like to execute the
files push
command to upload file(s) or/and folder(s) from my local machine to a remote server (dotCMS) by specifying the location of the local directory or file.Examples:
This command should consume the endpoint implemented here: #24738
Acceptance Criteria
Proposed Objective
Core Features
Proposed Priority
Priority 3 - Average
The text was updated successfully, but these errors were encountered: