Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to implement Multiple Progress Bars??? #31

Closed
marcellodesales opened this issue Dec 13, 2014 · 20 comments
Closed

How to implement Multiple Progress Bars??? #31

marcellodesales opened this issue Dec 13, 2014 · 20 comments

Comments

@marcellodesales
Copy link
Contributor

Problem

Hi there,

I have a list of files being downloaded by different go routines...

        for i := 0; i < len(urls); i++ {
                go func() { c <- download(urls[i])() }()
        }

I added the pb() implementation and it shows the progress bar, but it only displays in the same line...

Feature Request

  • Is there a way to control which line the progress bar will be printed?
  • If this is possible, I'd like to suggest the method "SetIndex" to set which line the progress must be printed after the current position on the terminal...
  • Any suggestions to change the Format method to do that?
// Sets the index of the progress, which gives the line position after the current terminal position.
func (pb *ProgressBar) SetIndex(index int) {}

This is similar to Docker's progress bars while pulling images...

[boot-1.pkg] 3.66 MB / 128.94 MB [==>------------------------------------------------] 2.84 % 759.61 KB/s 2m48s
[boot-2.pkg] 23.66 MB / 48.94 MB [===================>-------------------] 45.84 % 629.61 KB/s 2m48s
[boot-3.pkg] 30.00 MB / 300.94 MB [=====>------------------------------------------] 10.84 % 459.61 KB/s 2m48s

thanks!
Marcello

@cheggaaa
Copy link
Owner

Hi!
I made commit with experimental support
It's based on ANSI Escape Sequences
Checkout this branch https://github.com/cheggaaa/pb/tree/multiple

Example

package main

import (
    "math/rand"
    "pb"
    "sync"
    "time"
)

func main() {
    pool := &pb.Pool{}
    first := pb.New(1000).Prefix("First ")
    second := pb.New(1000).Prefix("Second ")
    third := pb.New(1000).Prefix("Third ")
    pool.Add(first, second, third)
    pool.Start()
    wg := new(sync.WaitGroup)
    for _, bar := range []*pb.ProgressBar{first, second, third} {
        wg.Add(1)
        go func(cb *pb.ProgressBar) {
            for n := 0; n < 1000; n++ {
                cb.Increment()
                time.Sleep(time.Millisecond * time.Duration(rand.Intn(100)))
            }
            cb.Finish()
            wg.Done()
        }(bar)
    }
    wg.Wait()
}

now have some problems, but i'm working on it

@marcellodesales
Copy link
Contributor Author

Hi @cheggaaa...

Thanks a lot for making this!!! The example works... Other suggestions to improve it:

  • Block any terminal key to be ENTERED other than Ctrl+C
  • Add this as an example in the examples directory

This is a very valuable component! Let me know how I can help... I will work on integrating the example for multiple files uploads... Let me know when you integrate this!

thanks!

marcellodesales added a commit to marcellodesales/pb that referenced this issue Dec 17, 2014
This commit adds the documentation for the README.md, along with the
description of the result of running the example.

*	modified:   README.md
- Some refactoring
- Adding the example

*	new file:   example/multiple.go
- Adding the example provided during the discusion of issue 31
- Adding the correct import statement for "pb"
cheggaaa added a commit that referenced this issue Dec 17, 2014
Issue #31: Updating the Docs with Parallel Bars example
@cheggaaa
Copy link
Owner

Block any terminal key to be ENTERED other than Ctrl+C

This not easy.. want to do this without heavy dependencies like ncurses.
If we'll know absolute cursor position, we can change only necessary rows without locking normal input and output. Until i can't found solution of this problem.

@marcellodesales
Copy link
Contributor Author

@cheggaaa That's alright! That's definitely a stretch...

I will try to create an example for the Multiple IO downloads, which is what I need... I tried last night but I might have missed something...

Thanks a lot for this!!! Easier than the others...

@cheggaaa
Copy link
Owner

Good news!
Now we have locked input - b2aef60

Bad news - windows is not yet supported

@cheggaaa
Copy link
Owner

Needs to be done:

  • win support
  • sortable bars
  • test on BSD and Solaris systems (i have only mac and linux)

@marcellodesales
Copy link
Contributor Author

@cheggaaa I was able to implement the IO-based Progress Bars with the sample you have (this is a real app)...

screen shot 2014-12-17 at 1 45 08 pm

I can contribute with the code sample code based on https://github.com/thbar/golang-playground/blob/master/download-files.go.

Wow, the locked input will definitely be great!!! :D I still have to test this on CI/CD environment (Travis and Jenkins for that matter)...

Other than that, I'm glad my team is not on Windows :D sorry for the windows users... Hopefully someone will tackle that... In addition, I like your TODOs... I can't test neither on BSD nor Solaris (same situation)...

Awesome! I'm so happy I got this working... I will work on the example code after tonight!!!

@marcellodesales
Copy link
Contributor Author

Hi @cheggaaa... Here's a feedback with the use of the Terminal: It helps with blocking the console, but it creates TOO MANY scrolls!!! That was NOT the plan :(

It is re-writing all the lines upon an update... I'd love to go back to square one, and be able to quit the process before... I had to close the window in order to stop the download of too many files and the experience was annoying...

Anyway, this is just my 2cs...

@marcellodesales
Copy link
Contributor Author

Not sure, I first thought that the commit 9ba607e worked... And I did not introduce any change to the library during my pull request...

Your initial support worked as expected, where there's no buffer in the terminal nor the scroll grows over the updated values. However, I was tagging the commits, merging to my local fork, and the same scrolling behavior showed up again...

Do you know why? The screenshot below shows the behavior of the terminal scrolling down instead of updating the status in-place, when compared to the first screenshot above.

screen shot 2014-12-18 at 7 21 58 am

@marcellodesales
Copy link
Contributor Author

Reason for the problem

if the number of PBs is greater than the number of lines in the current terminal window, then the scrolls will be engaged because of reprinting... Maximizing the terminal window to accommodate all my entries worked...

Possible Workaround

If there's no way to solve this, Is there a way to get the total number of lines of the terminal? That way, the PB API could expose the dimensions of the terminal and enqueue only the maximum number of PBs to be running. Further, it would be a matter of enqueuing the remainder while PBs are finished...

@cheggaaa
Copy link
Owner

About ctrl+c. We need copy some code from this package https://github.com/golang/crypto/tree/master/ssh/terminal and write method 'lockInput' (like this https://github.com/golang/crypto/blob/master/ssh/terminal/util.go#L87 , but without reading from stdin).
I'm doing it now.

About many bars. I think user has no need more than 5-6 bars) and problems like queues should solve user.

@cheggaaa
Copy link
Owner

Can you check last version? Under linux signals works fine

@marcellodesales
Copy link
Contributor Author

Well, the number of bars is up to the API Client to decide... What I'm suggesting is that if there are more bars than the terminal's current size can accommodate, then the PB API should give that information to the user's API... That way, users can use "Tickle" http://www.tecmint.com/manage-and-limit-downloadupload-bandwidth-with-trickle-in-linux/ to control the download bandwidth, or control the number of bars.

I will verify the last version in a few hours...

@uetchy
Copy link

uetchy commented Jul 3, 2015

Any updates?
I guess that it looks good to merge multiple branch into master.

@cheggaaa
Copy link
Owner

cheggaaa commented Jul 5, 2015

So far it's alpha.. We have not windows support and have some problems with catching os signals

@uetchy
Copy link

uetchy commented Jul 6, 2015

Did you checked out go-keybind ? This library can catch user inputs and supports windows.

marcellodesales added a commit to marcellodesales/go-nexus-artifacts-downloader that referenced this issue Oct 27, 2015
As requested at cheggaaa/pb#31, this commit
adds support to parallel progress bars that monitors the download of
multiple files from Nexus. This is similar to Docker's progress bars.

*	modified:   nexus/download.go
- Adding the library pb
- Adding documentation
- Adding the struct UrlMetadata that is used during the HTTP HEAD
  operation to retrieve the length of the resources.
- Adding the struct UrlDownload, which has the pointer to the
  UrlMetadata and the ProgressBar
- Adding the struct ResourceSizeDownload, which returns a function that
  returns the UrlMetadata
- Adding the function "collectBarsIndex" will retrieve the length of the
  files by executing an HTTP HEAD request to build the progress bars in
  Parallel using the channel. It times out in 1min just in case.
- Adding the function retrieveResourceLenth, that executes the HTTP HEAD
  for the URL and builds the UrlMetadata.
- Refactoring the function fanInDownloads to retrieve the progress bars
  index, create the progress bar pool. After that, it makes the request
to nexusDownload in parallel to download the urls.
- Removing the old progress channel implementation.
- Refactoring the method nexusDownload to take the UrlDownload struct
  pointer, which carries information about if the URL is broken. If not,
it continues with the download and uses the Multi-writer for the
progress bar, as described at
https://github.com/cheggaaa/pb/tree/multiple#progress-bar-for-io-operations
- Refactoring DownloadAllList to remove the progress Channel.
marcellodesales added a commit to marcellodesales/go-nexus-artifacts-downloader that referenced this issue Oct 27, 2015
…usted + Last Download to 100%

This commit refacts the download and progress to properly handle the
last download by waiting for 1sec after the process finishes. It
properly fills out the total download.

Finally, changing the progress bar library to use the forked version
because the current version imposes the locked-terminal. Until it is
resolved in the master branch, I will keep my fork.

cheggaaa/pb#31

*	modified:   nexus/download.go
- Removing the dependency to the progress libraries, refactored to the
  progress.go.
- Retrieving the instance of the pool
- Waiting 1sec to finish the download

*	modified:   nexus/progress.go
- Using my personal forked version of the PB library until the
  locked-terminal version gives the API to choose to not block the
  terminal.
- Updating the docs
- Fixing the prefix to be "[ obj.ext ]"
- Creating factory methods to properly create the library's version.
- Creating the pool based on the parameters.
@GideonRed-zz
Copy link

Any chance you can merge some minimal support for multiple progress bars into master? Doesn't have to satisfy every requirement set in this issue. Just something to give a few progress bars.

I can create the PR for you if you wish.

@cheggaaa
Copy link
Owner

cheggaaa commented Jan 7, 2016

Merged!
Code was cleaned and added some safety changes.
But anyway do not work at windows.

@Quentin-M
Copy link

Merged just in time!
We planned to start using it today on the experimental branch.
Thanks!

👍

@mnvx
Copy link

mnvx commented Apr 17, 2020

@cheggaaa Thanks for great feature and example. Would be great to add pbPool.Stop() into end of example:

package main

import (
    "math/rand"
    "pb"
    "sync"
    "time"
)

func main() {
    pool := &pb.Pool{}
    first := pb.New(1000).Prefix("First ")
    second := pb.New(1000).Prefix("Second ")
    third := pb.New(1000).Prefix("Third ")
    pool.Add(first, second, third)
    pool.Start()
    wg := new(sync.WaitGroup)
    for _, bar := range []*pb.ProgressBar{first, second, third} {
        wg.Add(1)
        go func(cb *pb.ProgressBar) {
            for n := 0; n < 1000; n++ {
                cb.Increment()
                time.Sleep(time.Millisecond * time.Duration(rand.Intn(100)))
            }
            cb.Finish()
            wg.Done()
        }(bar)
    }
    wg.Wait()

    // Stop pool, else keyboard will be locked after execution (as minimum in Ubuntu)
    pbPool.Stop()
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants