Skip to content

Commit

Permalink
[Frontend] Task server frontend changes. (#3673)
Browse files Browse the repository at this point in the history
[Frontend] Added a task server page, tasks table. desktop document clean up works (have to move it to celery though). Need to add a django db table to store the tasks table browser

Change-Id: I661b26388ffca4c42963b3ffcdf20f13c9914fa6
(cherry picked from commit 9a17b99)

[Frontend] 1. Added a django db table.  2. React fetches the results from this db table. 3. Added migrations scripts for the newly created task-models.py - django table

Change-Id: I7ae105083c0a5bac884c59b34cd801f5dee23f83
(cherry picked from commit 2287ffe)

[Frontend] Adding styling changes for task browser tables and adding label/icons

Change-Id: I3ee803fdaf60f808e5eb62151f15a94cc678561f
(cherry picked from commit 4d089f1)

[Frontend] Incomplete changes - greyout/reload file upload page, desktop_document_cleanup using redis

Change-Id: Ib53a9b81000a6c488b079bbcce617f8986ff3880
(cherry picked from commit 9fbdf55)

[Frontend] Fixing the upload popup, the stay until all uploads are complete. Fixing document clean up install dir and duration calculation

Change-Id: Ibf28fc12a05d4160a83ae57f727735daa3d8d6e7
(cherry picked from commit 403e990)

[Frontend] schedule /tmp clean_up job in task_server page

Change-Id: Id9d4ec893674c5df350bce3dbb8d4d21c254d1ce
(cherry picked from commit 32332da)

[Frontend] schedule /tmp clean_up job in task_server page

Change-Id: I2c039e46c1b81a9fc2df55301b9d3378338fbc16
(cherry picked from commit 366bcaa)

[Frontend] Changing progress Filter buttons (success, failure, running) to checkboxes

Change-Id: Iddbf0a687623e0bd598d5acd85201ace97510b36
(cherry picked from commit 006e14b)

[Frontend] Adding schedule upload support. Cleaned up default fallbackcomponent. Removed the csrf exempt annotations

Change-Id: I03b5ce1d0aba14fad50f0b41d14d71d1a4b8cf84
(cherry picked from commit 5d562a7)

[Frontend] /tmp cleaner runs every 1000 seconds using celery-beat scheduler. Files written onto /tmp folder in the last 15 minutes wont be deleted. This time interval ensures that when a user/service is writing onto /tmp, the files are not deleted.

Change-Id: Ic6b31173cc9a778da0c055a18435bdafc7384141
(cherry picked from commit cb0d1a0)

[Frontend] Adding support for killing celery tasks

Change-Id: Iba42d5f1d3c36dfddeeb8add2dcfa39b158b50d2
(cherry picked from commit 927d959)

[Frontend] Restricting uploads based on free space available in /tmp and MAX_FILE_SIZE_UPLOAD_LIMIT

Change-Id: I42d6d7a379dbd5b655be6b0e01c859d0b51294b3
(cherry picked from commit 8f4819d)

[Frontend] Improving schedule task popup and adding default values for tasks

Change-Id: If2e5ac784d338f479715741fc2ea74d97ad48b53
(cherry picked from commit 3f63e1e)

Improving styling using Antd and CUIX, pollForRealProgress. Cleaning up comments.

Change-Id: Ifdd00a48d366a0b845b0682f6d20df8f08f8e91f
(cherry picked from commit f7af6a6)

Improving styling using Antd and CUIX, pollForRealProgress. Cleaning up comments.

Change-Id: Ifdd00a48d366a0b845b0682f6d20df8f08f8e91f
(cherry picked from commit d01c62e)

Improving styling using Antd and CUIX, pollForRealProgress. Cleaning up comments.

Change-Id: Ifdd00a48d366a0b845b0682f6d20df8f08f8e91f
(cherry picked from commit ccf28ba)

Improving styling using Antd and CUIX, pollForRealProgress. Cleaning up comments.

Change-Id: Ifdd00a48d366a0b845b0682f6d20df8f08f8e91f
(cherry picked from commit e88f14c)

Only schedule upload button is enabled when task server is configured. Added tool tip for schedule upload button.

Change-Id: I4fb0dbd5626cd717ded3a1a3a1c40f0232b73067

improving log popup modal and url routing

Change-Id: Idb7637de564392142f6eb39bd58c5a8ee7fbe5dc

Co-authored-by: Athithyaa Selvam <[email protected]>
  • Loading branch information
athithyaaselvam and Athithyaa Selvam authored Apr 16, 2024
1 parent 03fcee0 commit 1cd19ea
Show file tree
Hide file tree
Showing 12 changed files with 919 additions and 10 deletions.
25 changes: 22 additions & 3 deletions apps/filebrowser/src/filebrowser/templates/listdir.mako
Original file line number Diff line number Diff line change
Expand Up @@ -164,21 +164,40 @@ ${ fb_components.menubar() }
<div class="btn-toolbar" style="display: inline; vertical-align: middle">
% if show_upload_button:
<!-- ko if: isS3 -->
<a class="btn fileToolbarBtn" title="${_('Upload files')}" data-bind="visible: !inTrash(), css: {'disabled': isS3Root()}, click: function(){ if (!isS3Root()) { uploadFile() }}"><i class="fa fa-arrow-circle-o-up"></i> ${_('Upload')}</a>
<!-- ko ifnot: isTaskServerEnabled -->
<a class="btn fileToolbarBtn" title="${_('Upload files')}" data-bind="visible: !inTrash(), css: {'disabled': isS3Root()}, click: function(){ if (!isS3Root()) { uploadFile(false) }}"><i class="fa fa-arrow-circle-o-up"></i> ${_('Upload')}</a>
<!-- /ko -->
<!-- ko if: isTaskServerEnabled -->
<a class="btn fileToolbarBtn" title="${_('Select a file to upload. The file will first be saved locally and then automatically transferred to the designated file system (e.g., S3, Azure) in the background. The upload modal closes immediately after the file is queued, allowing you to continue working. A notification, \'File upload scheduled. Please check the task server page for progress,\' will confirm the upload has started. This feature is especially useful for large files, as it eliminates the need to wait for the upload to complete.')}" data-bind="visible: !inTrash(), css: {'disabled': isS3Root()}, click: function(){ if (!isS3Root()) { uploadFile(true) }}"><i class="fa fa-arrow-circle-o-up"></i> ${_('Schedule Upload')}</a>
<!-- /ko -->
<!-- /ko -->
<!-- ko if: isGS -->
<a class="btn fileToolbarBtn" title="${_('Upload files')}" data-bind="visible: !inTrash(), css: {'disabled': isGSRoot()}, click: function(){ if (!isGSRoot()) { uploadFile() }}"><i class="fa fa-arrow-circle-o-up"></i> ${_('Upload')}</a>
<!-- /ko -->
<!-- ko if: isABFS -->
<a class="btn fileToolbarBtn" title="${_('Upload files')}" data-bind="visible: !inTrash(), css: {'disabled': isABFSRoot()}, click: function(){ if (!isABFSRoot()) { uploadFile() }}"><i class="fa fa-arrow-circle-o-up"></i> ${_('Upload')}</a>
<!-- ko ifnot: isTaskServerEnabled -->
<a class="btn fileToolbarBtn" title="${_('Upload files')}" data-bind="visible: !inTrash(), css: {'disabled': isABFSRoot()}, click: function(){ if (!isABFSRoot()) { uploadFile(false) }}"><i class="fa fa-arrow-circle-o-up"></i> ${_('Upload')}</a>
<!-- /ko -->
<!-- ko if: isTaskServerEnabled -->
<a class="btn fileToolbarBtn" title="${_('Select a file to upload. The file will first be saved locally and then automatically transferred to the designated file system (e.g., S3, Azure) in the background. The upload modal closes immediately after the file is queued, allowing you to continue working. A notification, \'File upload scheduled. Please check the task server page for progress,\' will confirm the upload has started. This feature is especially useful for large files, as it eliminates the need to wait for the upload to complete.')}" data-bind="visible: !inTrash(), css: {'disabled': isABFSRoot()}, click: function(){ if (!isABFSRoot()) { uploadFile(true) }}"><i class="fa fa-arrow-circle-o-up"></i> ${_('Schedule Upload')}</a>
<!-- /ko -->
<!-- /ko -->
<!-- ko ifnot: isS3() || isGS() || isABFS() -->
<!-- ko ifnot: isTaskServerEnabled -->
<div id="upload-dropdown" class="btn-group" style="vertical-align: middle">
<a data-hue-analytics="filebrowser:upload-btn-click" href="javascript: void(0)" class="btn upload-link dropdown-toggle" title="${_('Upload')}" data-bind="click: uploadFile, visible: !inTrash(), css: {'disabled': (isOFS() && (isOFSRoot() || isOFSServiceID() || isOFSVol()))}">
<a data-hue-analytics="filebrowser:upload-btn-click" href="javascript: void(0)" class="btn upload-link dropdown-toggle" title="${_('Upload')}" data-bind="click: function() { uploadFile(false); }, visible: !inTrash(), css: {'disabled': (isOFS() && (isOFSRoot() || isOFSServiceID() || isOFSVol()))}">
<i class="fa fa-arrow-circle-o-up"></i> ${_('Upload')}
</a>
</div>
<!-- /ko -->
<!-- ko if: isTaskServerEnabled -->
<div id="upload-dropdown" class="btn-group" style="vertical-align: middle">
<a data-hue-analytics="filebrowser:upload-btn-click" href="javascript: void(0)" class="btn upload-link dropdown-toggle" title="${_('Select a file to upload. The file will first be saved locally and then automatically transferred to the designated file system (e.g., S3, Azure) in the background. The upload modal closes immediately after the file is queued, allowing you to continue working. A notification, \'File upload scheduled. Please check the task server page for progress,\' will confirm the upload has started. This feature is especially useful for large files, as it eliminates the need to wait for the upload to complete.')}" data-bind="click: function() { uploadFile(true); }, visible: !inTrash(), css: {'disabled': (isOFS() && (isOFSRoot() || isOFSServiceID() || isOFSVol()))}">
<i class="fa fa-arrow-circle-o-up"></i> ${_('Schedule Upload')}
</a>
</div>
<!-- /ko -->
<!-- /ko -->
% endif
<div class="btn-group" style="vertical-align: middle">
<a href="javascript: void(0)" data-toggle="dropdown" class="btn dropdown-toggle" data-bind="visible: !inTrash(), css: {'disabled': isOFSRoot()}">
Expand Down
169 changes: 163 additions & 6 deletions apps/filebrowser/src/filebrowser/templates/listdir_components.mako
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ from django.template.defaultfilters import urlencode, stringformat, filesizeform
from desktop.lib.django_util import reverse_with_get, extract_field_data
from django.utils.encoding import smart_str
from filebrowser.conf import ENABLE_EXTRACT_UPLOADED_ARCHIVE, FILE_UPLOAD_CHUNK_SIZE, CONCURRENT_MAX_CONNECTIONS
from filebrowser.conf import ENABLE_EXTRACT_UPLOADED_ARCHIVE, FILE_UPLOAD_CHUNK_SIZE, CONCURRENT_MAX_CONNECTIONS, MAX_FILE_SIZE_UPLOAD_LIMIT
if sys.version_info[0] > 2:
from django.utils.translation import gettext as _
Expand Down Expand Up @@ -1109,6 +1109,10 @@ else:
return self.currentPath().toLowerCase().indexOf('ofs://') === 0;
});
self.isTaskServerEnabled = ko.computed(function() {
return window.getLastKnownConfig().hue_config.enable_chunked_file_uploader && window.getLastKnownConfig().hue_config.enable_task_server;
});
self.scheme = ko.pureComputed(function () {
var path = self.currentPath();
return path.substring(0, path.indexOf(':/')) || "hdfs";
Expand Down Expand Up @@ -2061,9 +2065,161 @@ else:
});
};
self.uploadFile = (function () {
var uploader;
if (window.getLastKnownConfig().hue_config.enable_chunked_file_uploader) {
function pollForTaskProgress(taskId, listItem, fileName) {
var taskStatus = 'pending';
var pollingInterval = 10000;
var doPoll = function() {
if (taskStatus === 'pending') {
$.get('/desktop/api2/taskserver/check_upload_status/' + taskId, function(data) {
if (data.isFinalized || data.isFailure || data.is_revoked) {
taskStatus = data.isFinalized ? 'finalized' : 'failed';
if (data.isFinalized) {
listItem.find('.progress-row-bar').css('width', '100%');
listItem.find('.progress-row-text').text('Upload complete.');
$(document).trigger('info', fileName + "${ _(' uploaded successfully.') }");
self.retrieveData(true);
} else if (data.isFailure) {
listItem.find('.progress-row-bar').css('width', '100%');
listItem.find('.progress-row-text').text('Upload failed.');
$(document).trigger('error', fileName + "${ _(' file upload failed. Please check the logs for task id: ') }" + taskId);
}
} else if (data.isRunning) {
var progressPercentage = 90; // Adjust based on data.progress if available
listItem.find('.progress-row-bar').css('width', progressPercentage + '%');
setTimeout(doPoll, pollingInterval);
}
}).fail(function(xhr, textStatus, errorThrown) {
if (xhr.status === 404) {
setTimeout(doPoll, pollingInterval); // Retry after 10 seconds
}
});
}
};
self.retrieveData(true);
doPoll();
}
self.uploadFile = (function () {
var uploader;
var scheduleUpload;
if ((window.getLastKnownConfig().hue_config.enable_chunked_file_uploader) && (window.getLastKnownConfig().hue_config.enable_task_server)) {
self.pendingUploads(0);
var action = "/filebrowser/upload/chunks/";
self.taskIds = [];
self.listItems = [];
uploader = new qq.FileUploader({
element: document.getElementById("fileUploader"),
request: {
endpoint: action,
paramsInBody: false,
params: {
dest: self.currentPath(),
inputName: "hdfs_file"
}
},
maxConnections: window.CONCURRENT_MAX_CONNECTIONS || 5,
chunking: {
enabled: true,
concurrent: {
enabled: true
},
partSize: window.FILE_UPLOAD_CHUNK_SIZE || 5242880,
success: {
endpoint: "/filebrowser/upload/complete/"
},
paramNames: {
partIndex: "qqpartindex",
partByteOffset: "qqpartbyteoffset",
chunkSize: "qqchunksize",
totalFileSize: "qqtotalfilesize",
totalParts: "qqtotalparts"
}
},
template: 'qq-template',
callbacks: {
onProgress: function (id, fileName, loaded, total) {
$('.qq-upload-files').find('li').each(function(){
var listItem = $(this);
if (listItem.find('.qq-upload-file-selector').text() == fileName){
//cap the progress at 80%
listItem.find('.progress-row-bar').css('width', (loaded/total)*80 + '%');
if ((loaded/total) === 80) {
listItem.find('.progress-row-text').text('Finalizing upload...');
}
}
});
},
onComplete: function (id, fileName, response) {
self.pendingUploads(self.pendingUploads() - 1);
if (response.status != 0) {
huePubSub.publish('hue.global.error', {message: "${ _('Error: ') }" + response.data});
}
else {
var task_id = response.task_id;
self.taskIds.push(task_id);
var listItem = $('.qq-upload-files').find('li').filter(function() {
return $(this).find('.qq-upload-file-selector').text() === fileName;
});
self.listItems.push(listItem);
if (scheduleUpload && self.pendingUploads() === 0) {
$('#uploadFileModal').modal('hide');
$(document).trigger('info', "File upload scheduled. Please check the task server page for progress.");
}
pollForTaskProgress(response.task_id, listItem, fileName);
self.filesToHighlight.push(response.path);
}
if (self.pendingUploads() === 0) {
self.taskIds=[];
self.listItems=[];
self.retrieveData(true);
}
},
onSubmit: function (id, fileName, responseJSON) {
var deferred = new qq.Promise(); // Create a promise to defer the upload
var uploader = this;
// Make an AJAX request to check available disk space
$.ajax({
url: '/desktop/api2/taskserver/get_available_space/',
success: function(response) {
var freeSpace = response.free_space;
var file = uploader.getFile(id); // Use the stored reference
if ((file.size > freeSpace) && (file.size > window.MAX_FILE_SIZE_UPLOAD_LIMIT*1024*1024)) {
$(document).trigger('info', "Not enough space available to upload this file.")
deferred.failure(); // Reject the promise to cancel the upload
} else {
var newPath = "/filebrowser/upload/chunks/file?dest=" + encodeURIComponent(self.currentPath().normalize('NFC'));
uploader.setEndpoint(newPath);
self.pendingUploads(self.pendingUploads() + 1);
deferred.success(); // Resolve the promise to allow the upload
}
},
error: function(xhr, status, error) {
alert('Error checking available space: ' + error);
deferred.failure(); // Reject the promise to cancel the upload
}
});
return deferred; // Return the promise to Fine Uploader
},
onCancel: function (id, fileName) {
self.pendingUploads(self.pendingUploads() - 1);
}
},
debug: false
});
}
// Chunked Fileuploader without Taskserver
else if ((window.getLastKnownConfig().hue_config.enable_chunked_file_uploader) && !(window.getLastKnownConfig().hue_config.enable_task_server)) {
self.pendingUploads(0);
var action = "/filebrowser/upload/chunks/";
uploader = new qq.FileUploader({
Expand Down Expand Up @@ -2098,7 +2254,6 @@ else:
template: 'qq-template',
callbacks: {
onProgress: function (id, fileName, loaded, total) {
console.log(loaded);
$('.qq-upload-files').find('li').each(function(){
var listItem = $(this);
if (listItem.find('.qq-upload-file-selector').text() == fileName){
Expand Down Expand Up @@ -2137,6 +2292,7 @@ else:
debug: false
});
}
//Regular fileuploads
else {
self.pendingUploads(0);
var action = "/filebrowser/upload/file";
Expand Down Expand Up @@ -2211,7 +2367,8 @@ else:
});
});
return function () {
return function (isScheduled) {
scheduleUpload = isScheduled;
$("#uploadFileModal").modal({
show: true
});
Expand Down
6 changes: 6 additions & 0 deletions desktop/core/src/desktop/js/onePageViewModel.js
Original file line number Diff line number Diff line change
Expand Up @@ -558,6 +558,12 @@ class OnePageViewModel {
});
}
},
{
url: '/task_server',
app: function () {
self.loadApp('taskserver');
}
},
{
url: '/desktop/connectors',
app: function () {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,4 +12,4 @@ const FallbackComponent = () => {
);
};

export default FallbackComponent;
export default FallbackComponent;
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
@use 'variables' as vars;
@import '@cloudera/cuix-core/variables.scss';

.log-content-scrollable {
max-height: 600px;
overflow-y: auto;
margin-right: -16px;
padding-right: 16px;
}

.customCursorClass {
cursor: pointer;
}

.flex-container-style {
display: flex;
align-items: center;
gap: 10px;
}

.task-selection {
margin-bottom: 2rem;
}

.task-dropdown {
width: 100%;
padding: 0.5rem;
font-size: 1rem;
}

.parameter-inputs {
display: flex;
flex-direction: column;
gap: 1rem;
margin-top: 2rem;
}

.parameter-row {
display: flex;
align-items: center;
}

.parameter-label {
flex: 1;
width: 100px;
font-weight: bold;
margin-right: 1rem;
}

.parameter-input {
padding: 0.5rem;
flex: 1;
font-size: 1rem;
width: 100%;
}

.vertical-spacer {
height: 2rem;
}

.row-success {
border-left: 4px solid $fluidx-green-500;
}

.row-failure {
border-left: 4px solid $fluidx-red-500;
}

.row-running {
border-left: 4px solid $fluidx-amber-500;
}

.ant-btn-primary {
background-color: $fluidx-blue-500;
border-color: $fluidx-blue-600;
}

.ant-btn-danger {
background-color: $fluidx-red-500;
border-color: $fluidx-red-600;
}

.ant-tag-success {
background-color: $fluidx-green-500;
color: $fluidx-gray-900;
}

.ant-tag-error {
background-color: $fluidx-red-500;
color: $fluidx-gray-900;
}

.ant-tag-warning {
background-color: $fluidx-amber-500;
color: $fluidx-gray-900;
}

.success-text {
color: $fluidx-green-500;
}

.running-text {
color: $fluidx-amber-500;
}

.failed-text {
color: $fluidx-red-500;
}
Loading

0 comments on commit 1cd19ea

Please sign in to comment.