Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cant quit sometimes #121

Closed
deliciouslytyped opened this issue Apr 11, 2021 · 12 comments
Closed

Cant quit sometimes #121

deliciouslytyped opened this issue Apr 11, 2021 · 12 comments
Assignees
Labels
bug Confirmed to be a bug please try it

Comments

@deliciouslytyped
Copy link

deliciouslytyped commented Apr 11, 2021

I'm on 2.2.0 .
Sometimes I can't quit. The confirmation dialog comes up but nothing happens when I click or hit enter on the ok button.
The cancel button works fine.

ps aux | grep tshark lists no tshark processes but the "waiting for packets" animation thing is still going.

I don't know how to repro this.

https://www.youtube.com/watch?v=7cbjlV1l28U

@gcla
Copy link
Owner

gcla commented Apr 11, 2021

Yikes again. That's bad. Please could you add the following to ~/.config/termshark/termshark.toml:

[main]
  debug = true

Now when you start termshark, it will run a Go debugging server on localhost:6060. When termshark is stuck and won't terminate, it would be very useful to see the contents of http://localhost:6060/debug/pprof/goroutine?debug=1

Thanks!

@gcla
Copy link
Owner

gcla commented Apr 11, 2021

I've assumed you're on Linux - let me know if not and you need help finding the config file.

@deliciouslytyped
Copy link
Author

Sounds good.

@deliciouslytyped
Copy link
Author

Here's a trace:

# curl http://localhost:6060/debug/pprof/goroutine?debug=1                                                
goroutine profile: total 20                                                                                            
4 @ 0x43c6c5 0x44d9b7 0x4b622b 0x4b6a0c 0x5c9069 0x5d4217 0x473ca1                                                     
#       0x4b622a        io.(*pipe).Read+0xca                                    io/pipe.go:57                          
#       0x4b6a0b        io.(*PipeReader).Read+0x4b                              io/pipe.go:134                         
#       0x5c9068        bufio.(*Scanner).Scan+0xa8                              bufio/scan.go:214                                                                                                                                             
#       0x5d4216        github.com/sirupsen/logrus.(*Entry).writerScanner+0xb6  github.com/sirupsen/[email protected]/writer.go:53                                                                                                                
                                                                                                                                                                                                                                              
1 @ 0x40e174 0x470325 0x5ad7c5 0x473ca1                                                                                                                                                                                                       
#       0x470324        os/signal.signal_recv+0xa4      runtime/sigqueue.go:168                                                                                                                                                               
#       0x5ad7c4        os/signal.loop+0x24             os/signal/signal_unix.go:23                                    

1 @ 0x43c6c5 0x4074fa 0x407255 0x8ecf75 0x8d8c25 0x473ca1  
#       0x8ecf74        github.com/gcla/termshark/v2/pcap.(*PdmlLoader).loadPcapSync.func5.6+0x494      github.com/gcla/termshark/v2/pcap/loader.go:1198
#       0x8d8c24        github.com/gcla/termshark/v2.TrackedGo.func1+0x84                               github.com/gcla/termshark/v2/utils.go:593

1 @ 0x43c6c5 0x434d1b 0x46df55 0x4ddb85 0x4dec75 0x4dec57 0x4e7b57 0x4e7b27 0x4b494c 0x51b145 0x51b113 0x51b1c7 0x473ca1
#       0x46df54        internal/poll.runtime_pollWait+0x54             runtime/netpoll.go:222                         
#       0x4ddb84        internal/poll.(*pollDesc).wait+0x44             internal/poll/fd_poll_runtime.go:87            
#       0x4dec74        internal/poll.(*pollDesc).waitRead+0x1d4        internal/poll/fd_poll_runtime.go:92            
#       0x4dec56        internal/poll.(*FD).Read+0x1b6                  internal/poll/fd_unix.go:166                   
#       0x4e7b56        os.(*File).read+0x76                            os/file_posix.go:31                            
#       0x4e7b26        os.(*File).Read+0x46                            os/file.go:117                                 
#       0x4b494b        io.copyBuffer+0x12b                             io/io.go:423                                   
#       0x51b144        io.Copy+0x64                                    io/io.go:382                                   
#       0x51b112        os/exec.(*Cmd).writerDescriptor.func1+0x32      os/exec/exec.go:311                            
#       0x51b1c6        os/exec.(*Cmd).Start.func1+0x26                 os/exec/exec.go:441                            

1 @ 0x43c6c5 0x434d1b 0x46df55 0x4ddb85 0x4dec75 0x4dec57 0x4e7b57 0x4e7b27 0x5c1646 0x473ca1                          
#       0x46df54        internal/poll.runtime_pollWait+0x54                     runtime/netpoll.go:222                 
#       0x4ddb84        internal/poll.(*pollDesc).wait+0x44                     internal/poll/fd_poll_runtime.go:87    
#       0x4dec74        internal/poll.(*pollDesc).waitRead+0x1d4                internal/poll/fd_poll_runtime.go:92    
#       0x4dec56        internal/poll.(*FD).Read+0x1b6                          internal/poll/fd_unix.go:166           
#       0x4e7b56        os.(*File).read+0x76                                    os/file_posix.go:31                    
#       0x4e7b26        os.(*File).Read+0x46                                    os/file.go:117                         
#       0x5c1645        github.com/gdamore/tcell.(*tScreen).inputLoop+0xc5      github.com/gdamore/[email protected]/tscreen.go:1355

1 @ 0x43c6c5 0x434d1b 0x46df55 0x4ddb85 0x4dec75 0x4dec57 0x67facf 0x68f1b1 0x825858 0x473ca1                          
#       0x46df54        internal/poll.runtime_pollWait+0x54             runtime/netpoll.go:222                         
#       0x4ddb84        internal/poll.(*pollDesc).wait+0x44             internal/poll/fd_poll_runtime.go:87            
#       0x4dec74        internal/poll.(*pollDesc).waitRead+0x1d4        internal/poll/fd_poll_runtime.go:92            
#       0x4dec56        internal/poll.(*FD).Read+0x1b6                  internal/poll/fd_unix.go:166                   
#       0x67face        net.(*netFD).Read+0x4e                          net/fd_posix.go:55                             
#       0x68f1b0        net.(*conn).Read+0x90                           net/net.go:183                                 
#       0x825857        net/http.(*connReader).backgroundRead+0x57      net/http/server.go:692                         

1 @ 0x43c6c5 0x434d1b 0x46df55 0x4ddb85 0x4e09f2 0x4e09d4 0x681105 0x699092 0x697e85 0x830f85 0x830c9a 0xa7f678 0xa7f642 0x473ca1
#       0x46df54        internal/poll.runtime_pollWait+0x54             runtime/netpoll.go:222                         
#       0x4ddb84        internal/poll.(*pollDesc).wait+0x44             internal/poll/fd_poll_runtime.go:87            
#       0x4e09f1        internal/poll.(*pollDesc).waitRead+0x211        internal/poll/fd_poll_runtime.go:92 
#       0x4e09d3        internal/poll.(*FD).Accept+0x1f3                internal/poll/fd_unix.go:401                   
#       0x681104        net.(*netFD).accept+0x44                        net/fd_unix.go:172                             
#       0x699091        net.(*TCPListener).accept+0x31                  net/tcpsock_posix.go:139                       
#       0x697e84        net.(*TCPListener).Accept+0x64                  net/tcpsock.go:261                             
#       0x830f84        net/http.(*Server).Serve+0x284                  net/http/server.go:2981                        
#       0x830c99        net/http.(*Server).ListenAndServe+0xb9          net/http/server.go:2910                        
#       0xa7f677        net/http.ListenAndServe+0x57                    net/http/server.go:3164                        
#       0xa7f641        main.cmain.func2+0x21                           github.com/gcla/termshark/v2/cmd/termshark/termshark.go:481                                                                                                           
                                                                                                                                                                                                                                              
1 @ 0x43c6c5 0x44d9b7 0x5bf488 0x612603 0x473ca1                                                                                                                                                                                              
#       0x5bf487        github.com/gdamore/tcell.(*tScreen).PollEvent+0x87              github.com/gdamore/[email protected]/tscreen.go:793                                                                          
#       0x612602        github.com/gcla/gowid.(*App).StartTCellEvents.func1+0x62        github.com/gcla/[email protected]/app.go:557                                                                                 
                                                                                                                       
1 @ 0x43c6c5 0x44d9b7 0x5c11bd 0x473ca1                    
#       0x5c11bc        github.com/gdamore/tcell.(*tScreen).mainLoop+0xfc       github.com/gdamore/[email protected]/tscreen.go:1301
                                                                                                                                                                                                                                              
1 @ 0x43c6c5 0x44d9b7 0x8d4b6b 0xa4247e 0x8d8c25 0x473ca1                                                                                                                                                                                     
#       0x8d4b6a        github.com/gcla/termshark/v2.RunOnDoubleTicker+0xaa                     github.com/gcla/termshark/v2/utils.go:570
#       0xa4247d        github.com/gcla/termshark/v2/ui.SetStructWidgets.BeforeBegin.func1+0x7d github.com/gcla/termshark/v2/ui/prochandlers.go:303
#       0x8d8c24        github.com/gcla/termshark/v2.TrackedGo.func1+0x84                       github.com/gcla/termshark/v2/utils.go:593
                                                                                                                       
1 @ 0x43c6c5 0x44d9b7 0x8d89b0 0x8d8c25 0x473ca1                                                                       
#       0x8d89af        github.com/gcla/termshark/v2.NewConfigWatcher.func1+0x10f       github.com/gcla/termshark/v2/confwatcher.go:50
#       0x8d8c24        github.com/gcla/termshark/v2.TrackedGo.func1+0x84               github.com/gcla/termshark/v2/utils.go:593
                                                                                                                       
1 @ 0x43c6c5 0x44d9b7 0xa03875 0x8d8c25 0x473ca1                                                                       
#       0xa03874        github.com/gcla/termshark/v2/widgets/filter.New.func5+0xf4      github.com/gcla/termshark/v2/widgets/filter/filter.go:244
#       0x8d8c24        github.com/gcla/termshark/v2.TrackedGo.func1+0x84               github.com/gcla/termshark/v2/utils.go:593
                                                                                                                       
1 @ 0x43c6c5 0x44d9b7 0xa03aa5 0x8d8c25 0x473ca1           
#       0xa03aa4        github.com/gcla/termshark/v2/widgets/filter.New.func6+0x184     github.com/gcla/termshark/v2/widgets/filter/filter.go:274
#       0x8d8c24        github.com/gcla/termshark/v2.TrackedGo.func1+0x84               github.com/gcla/termshark/v2/utils.go:593
                                                                                                                       
1 @ 0x43c6c5 0x44d9b7 0xa7a0ec 0xa77b1a 0x43c296 0x473ca1                                                              
#       0xa7a0eb        main.cmain+0x252b       github.com/gcla/termshark/v2/cmd/termshark/termshark.go:974            
#       0xa77b19        main.main+0x79          github.com/gcla/termshark/v2/cmd/termshark/termshark.go:62             
#       0x43c295        runtime.main+0x255      runtime/proc.go:225                                                    
                                                                                                                                                                                                                                              
1 @ 0x43c6c5 0x44e485 0x44e46e 0x46fb05 0x48cca5 0x8ea515 0x8ee3cf 0x8d8c25 0x473ca1                                   
#       0x46fb04        sync.runtime_Semacquire+0x44                                                    runtime/sema.go:56
#       0x48cca4        sync.(*WaitGroup).Wait+0x64                                                     sync/waitgroup.go:130
#       0x8ea514        github.com/gcla/termshark/v2/pcap.(*PdmlLoader).loadPcapSync.func5.2+0x34       github.com/gcla/termshark/v2/pcap/loader.go:805
#       0x8ee3ce        github.com/gcla/termshark/v2/pcap.(*PdmlLoader).loadPcapSync.func5+0x86e        github.com/gcla/termshark/v2/pcap/loader.go:1269
#       0x8d8c24        github.com/gcla/termshark/v2.TrackedGo.func1+0x84                               github.com/gcla/termshark/v2/utils.go:593
                                                                                                                       
1 @ 0x46db5d 0x5ebf8e 0x5ebd65 0x5e88f2 0xa75465 0xa76cb7 0x82d5e4 0x82f46d 0x830ba3 0x82c0cd 0x473ca1                 
#       0x46db5c        runtime/pprof.runtime_goroutineProfileWithLabels+0x5c   runtime/mprof.go:716                   
#       0x5ebf8d        runtime/pprof.writeRuntimeProfile+0xcd                  runtime/pprof/pprof.go:724             
#       0x5ebd64        runtime/pprof.writeGoroutine+0xa4                       runtime/pprof/pprof.go:684                                                                                                                                    
#       0x5e88f1        runtime/pprof.(*Profile).WriteTo+0x3f1                  runtime/pprof/pprof.go:331             
#       0xa75464        net/http/pprof.handler.ServeHTTP+0x384                  net/http/pprof/pprof.go:253            
#       0xa76cb6        net/http/pprof.Index+0x8d6                              net/http/pprof/pprof.go:371    
#       0x82d5e3        net/http.HandlerFunc.ServeHTTP+0x43                     net/http/server.go:2069                                                                                                                                       
#       0x82f46c        net/http.(*ServeMux).ServeHTTP+0x1ac                    net/http/server.go:2448                                                                                                                                       
#       0x830ba2        net/http.serverHandler.ServeHTTP+0xa2                   net/http/server.go:2887                                                                                                                                       
#       0x82c0cc        net/http.(*conn).serve+0x8cc                            net/http/server.go:1952                                                                                                                                       
                                                                                                                       
1 @ 0x4c3485 0x5ae672 0x61ed51 0x61df66 0x473ca1                                                                       
#       0x4c3484        syscall.Syscall6+0x4                                            syscall/asm_linux_amd64.s:43                                                                                                                          
#       0x5ae671        golang.org/x/sys/unix.EpollWait+0x71                            golang.org/x/[email protected]/unix/zsyscall_linux_amd64.go:1790                                                                 
#       0x61ed50        gopkg.in/fsnotify/fsnotify%2ev1.(*fdPoller).wait+0x90           gopkg.in/fsnotify/[email protected]/inotify_poller.go:86                                                                                             
#       0x61df65        gopkg.in/fsnotify/fsnotify%2ev1.(*Watcher).readEvents+0x205     gopkg.in/fsnotify/[email protected]/inotify.go:192 

@gcla
Copy link
Owner

gcla commented Apr 12, 2021

Thanks! That helps me understand where the problem lies. I'm trying now to come up with a theory for what gets termshark into this state. Is there anything notable about your use of the program when this happens e.g. are you reading from an interface? Or do you load up more than one pcap in the same termshark session? Would you mind attaching the log file too? It should be ~/.cache/termshark/termshark.log.

@gcla
Copy link
Owner

gcla commented Apr 12, 2021

I have a theory - I see a hole in my process handling. I'll try to get a binary to you to try, as a replacement.

gcla added a commit that referenced this issue Apr 12, 2021
This is an attempt to solve issue #121. I think my mistake is as
follows:

- I start the goroutine that runs the pcap process
- after starting the process, I send its pid via a channel to
  another goroutine that controls process life-cycles
- however that goroutine has closed down because the user issued
  a cancel

The process lifetime goroutine should not shut down if it is guaranteed
the pcap running goroutine is going to send its pid, because then
nothing will listen for it. I've adjusted the logic so now that
goroutine will definitely wait for the pid - unless it receives a pid==0
which indicates there was a problem starting the process.
@gcla
Copy link
Owner

gcla commented Apr 12, 2021

Hi @deliciouslytyped - I pushed a commit with an experimental fix. Would you be willing to try it? It would just mean replacing one binary - or just put this one in your PATH ahead of the one you're using now. The binary will still advertise itself as v2.2.0, but it's really v2.2.0 with one extra commit.

Here are the download links:

Linux: https://storage.googleapis.com/termshark/43216e2b97c71fba29d11945a5800e01f81e2d64/termshark_linux_amd64/termshark

macOS: https://storage.googleapis.com/termshark/43216e2b97c71fba29d11945a5800e01f81e2d64/termshark_darwin_amd64/termshark

Windows: https://storage.googleapis.com/termshark/43216e2b97c71fba29d11945a5800e01f81e2d64/termshark_windows_amd64/termshark.exe

@gcla gcla self-assigned this Apr 12, 2021
@gcla gcla added bug Confirmed to be a bug please try it labels Apr 12, 2021
@deliciouslytyped
Copy link
Author

deliciouslytyped commented Apr 12, 2021

I would strongly prefer a source patch on a branch. (the thing is, running random binaries on nixos is more of a pain anyway)
Edit: oh, I see the commit
So basically, my distro packages builds from source and I basically just need to change the rev to get a different version, but I appreciate the effort. I'll give it a poke later I guess, but how do I know if it's working?

@deliciouslytyped
Copy link
Author

deliciouslytyped commented Apr 12, 2021

I don't know about architecting this stuff, but I think stuff like handling (forced) exit shouldn't fail "stuck". Cleanup code should be reliable "by default" - i.e. even if it fails for unforseen reasons it should still more or less work in some intended way. - but I realize I'm being very handwavy.

@gcla
Copy link
Owner

gcla commented Apr 13, 2021

Understood - if you have the source code, the commit to build from is 43216e2; or git checkout loader-shutdown-bug and build that branch. go install ./... should be enough.

I take your point about failing stuck. I block at shutdown deliberately, but maybe it's the wrong choice for official releases. It gives me a better shot of being alerted to program flows that expose bugs, but I'm not sure if it's fair to users.

@deliciouslytyped
Copy link
Author

That doesn't seem unreasonable, maybe there's a better way to get the best of both worlds. If you can get what you want but keep it responsive that's probably fine for me.

@gcla
Copy link
Owner

gcla commented May 15, 2021

I think this is fixed as of bfea053. Please re-open if you see it come back!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Confirmed to be a bug please try it
Projects
None yet
Development

No branches or pull requests

2 participants