Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Block SIGPIPE on background thread #169

Merged
merged 2 commits into from
Aug 15, 2018
Merged

Block SIGPIPE on background thread #169

merged 2 commits into from
Aug 15, 2018

Conversation

wch
Copy link
Collaborator

@wch wch commented Aug 14, 2018

This (hopefully) fixes #168. @shapenaji, @atheriel, can you try installing this and see if it fixes rstudio/plumber#289?


Note: The pthread_sigmask() documentation wasn't clear about what exactly would happen if a blocked signal is sent to the thread -- would the signal be then sent to another available thread? I wrote the program below to test it out and found that no, the signal does not get sent to another thread (which is what we want).

#include <pthread.h>
#include <stdio.h>
#include <signal.h>


void block_sigpipe() {
  sigset_t set;
  int result;
  sigemptyset(&set);
  sigaddset(&set, SIGPIPE);
  fprintf(stderr, "Blocking SIGPIPE on thread.\n");
  result = pthread_sigmask(SIG_BLOCK, &set, NULL);
  if (result) {
    fprintf(stderr, "Error blocking SIGPIPE.\n");
  }
}

void* bg_func(void* data) {
  fprintf(stderr, "Started bg thread\n");

  // Comment out this line to see behavior if signal is not blocked
  block_sigpipe();

  fprintf(stderr, "Signalling SIGPIPE to thread... ");
  pthread_kill(pthread_self(), SIGPIPE);
  fprintf(stderr, "done\n");

  return NULL;
}

int main(int argc, char ** argv) {
  pthread_t bg_thread;

  if (pthread_create(&bg_thread, NULL, bg_func, NULL)) {
    fprintf(stderr, "Error creating thread.\n");
    return 1;
  }

  // Wait for thread to finish.
  if (pthread_join(bg_thread, NULL)) {
    fprintf(stderr, "Error joining thread.\n");
    return 2;
  }

  // If it gets here, then the SIGPIPE did not get sent to the main thread.
  fprintf(stderr, "Joined thread. Exiting.\n");

  return 0;
}

@atheriel
Copy link
Contributor

We'll test this out today. Thanks for your work on this!

@atheriel
Copy link
Contributor

OK, this branch seems to fix our problems, in that the process seems unable to receive SIGPIPE at all anymore.

However, I did some work of my own on this yesterday and thought I'd present another approach, using libuv's signal support instead of pthread_sigmask(). This might fit better with httpuv's overall design. It also isolates the problematic SIGPIPE handling in httpuv's background thread, instead of in the entire application. SIGPIPE is still sent to the thread, but it now calls its own (empty) handler instead of calling R's handler. In gdb it looks like this:

$ R-3.4.4/bin/R -d "gdb --silent" --no-save --no-restore -e "library(plumber);srv <- plumb(\"plumber.R\");srv\$run(port = 9550)"
Reading symbols from /opt/api/testapi/R-3.4.4/bin/exec/R...done.
(gdb) break main.c:handlePipe
No source file named main.c.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (main.c:handlePipe) pending.
(gdb) run
Starting program: /opt/api/testapi/R-3.4.4/bin/exec/R --no-save --no-restore -e library\(plumber\)\;srv\~+\~\<-\~+\~plumb\(\"plumber.R\"\)\;srv\$run\(port\~+\~=\~+\~9550\)
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

R version 3.4.4 (2018-03-15) -- "Someone to Lean On"
Copyright (C) 2018 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> library(plumber);srv <- plumb("plumber.R");srv$run(port = 9550)
Starting server to listen on port 9550
[New Thread 0x7ffff01f9700 (LWP 27785)]
[New Thread 0x7fffef9f8700 (LWP 27786)]

Thread 2 "R" received signal SIGPIPE, Broken pipe.
[Switching to Thread 0x7ffff01f9700 (LWP 27785)]
0x00007ffff75404bd in write () at ../sysdeps/unix/syscall-template.S:84
84	../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) c
Continuing.

Thread 2 "R" received signal SIGPIPE, Broken pipe.
0x00007ffff75404bd in write () at ../sysdeps/unix/syscall-template.S:84
84	in ../sysdeps/unix/syscall-template.S
(gdb) c
Continuing.

Thread 2 "R" received signal SIGPIPE, Broken pipe.
0x00007ffff75404bd in write () at ../sysdeps/unix/syscall-template.S:84
84	in ../sysdeps/unix/syscall-template.S
(gdb) c
Continuing.

As you can see, we are now handling the signals ourselves, and R no longer segfaults.

The patch I used to achieve this is below. If you like this approach I can submit it as a PR or you can git am the patch directly.

From c5c8b05e2707668cb5565ad18daac3748062ca88 Mon Sep 17 00:00:00 2001                                      
From: Aaron Jacobs <[email protected]>                                                   
Date: Tue, 14 Aug 2018 15:36:24 +0000                                                                       
Subject: [PATCH] Handle SIGPIPE via libuv. Fixes #168.                                                      
                                                                                                            
Signed-off-by: Aaron Jacobs <[email protected]>                                          
---                                                                                                         
 src/httpuv.cpp | 31 +++++++++++++++++++++++++++++++                                                        
 1 file changed, 31 insertions(+)                                                                           
                                                                                                            
diff --git a/src/httpuv.cpp b/src/httpuv.cpp                                                                
index 07053d5..614b57d 100644                                                                               
--- a/src/httpuv.cpp                                                                                        
+++ b/src/httpuv.cpp                                                                                        
@@ -43,6 +43,16 @@ public:                                                                                  
     guard guard(_mutex);                                                                                   
     if (!_initialized) {                                                                                   
       uv_loop_init(&_loop);                                                                                
+#if !defined(_WIN32)                                                                                       
+      // Suppress SIGPIPE.                                                                                 
+      int rc = uv_signal_init(&_loop, &_sigpipe_signal);                                                   
+      if (rc == 0) {                                                                                       
+        rc = uv_signal_start(&_sigpipe_signal, _on_sigpipe, SIGPIPE);                                      
+      }                                                                                                    
+      if (rc != 0) {                                                                                       
+        throw std::runtime_error("failed to start signal handler");                                        
+      }                                                                                                    
+#endif                                                                                                     
       _initialized = true;                                                                                 
     }                                                                                                      
   }                                                                                                        
@@ -56,6 +66,19 @@ public:                                                                                  
     return &_loop;                                                                                         
   };                                                                                                       
                                                                                                            
+  int stop_signal_handler() {                                                                              
+    guard guard(_mutex);                                                                                   
+    if (!_initialized) {                                                                                   
+      throw std::runtime_error("io_loop not initialized!");                                                
+    }                                                                                                      
+#if !defined(_WIN32)                                                                                       
+    // Halt SIGPIPE suppression.                                                                           
+    return uv_signal_stop(&_sigpipe_signal);                                                               
+#else                                                                                                      
    return 0;                                                                                              
+#endif                                                                                                     
+  }                                                                                                        
+                                                                                                           
   void reset() {                                                                                           
     guard guard(_mutex);                                                                                   
     _initialized = false;                                                                                  
@@ -65,6 +88,13 @@ private:                                                                                 
   uv_loop_t _loop;                                                                                         
   uv_mutex_t _mutex;                                                                                       
   bool _initialized;                                                                                       
+                                                                                                           
+#if !defined(_WIN32)                                                                                       
+  uv_signal_t _sigpipe_signal;                                                                             
+  static void _on_sigpipe(uv_signal_t* signal, int signum) {                                               
+    // Do nothing, i.e. suppress SIGPIPE.                                                                  
+  }                                                                                                        
+#endif                                                                                                     
 };                                                                                                         
                                                                                                            
 // ============================================================================                            
@@ -116,6 +146,7 @@ void io_thread(void* data) {                                                            
   trace("io_loop stopped");                                                                                
                                                                                                            
   // Cleanup stuff                                                                                         
+  io_loop.stop_signal_handler();                                                                           
   uv_walk(io_loop.get(), close_handle_cb, NULL);                                                           
   uv_run(io_loop.get(), UV_RUN_ONCE);                                                                      
   uv_loop_close(io_loop.get());                                                                            
--                                                                                                          
2.7.4

@wch
Copy link
Collaborator Author

wch commented Aug 14, 2018

@atheriel Your approach looks interesting... Are you sure that it will not affect how other threads in the process handle SIGPIPE?

We would consider a PR -- @jcheng5 and I will have to figure out which approach to go with. If you want to do a PR, please fill out the contributor agreement.

@wch
Copy link
Collaborator Author

wch commented Aug 15, 2018

One more note about this PR: it makes it so the httpuv background thread stops listening for SIGPIPE. However, the process overall can still receive SIGPIPE -- it just won't be handled by the background thread.

@atheriel
Copy link
Contributor

@wch After some testing, I can confirm that this PR only blocks SIGPIPE on the background threads. I can still trigger R's signal handler (which just raises an error) by sending the signal to the main thread explicitly.

On the other hand, my approach seems to take over signal handling for the entire process -- even if you send SIGPIPE to the main thread it will still be handled by httpuv. This surprised me (given libuv's documentation), so my earlier assessment of the patch was wrong.

I'm not sure which approach is more desirable.

@wch
Copy link
Collaborator Author

wch commented Aug 15, 2018

@atheriel It sounds like libuv's uv_signal_start isn't isolating to a thread, which is what I was worried about (since the documentation for it doesn't say anything about threads). I don't think we want to interfere with the SIGPIPE handler on R's main thread.

@jcheng5 Do you think we need to handle/block SIGPIPE for the later package as well?

@wch wch merged commit cb697db into master Aug 15, 2018
@wch wch deleted the fix-sigpipe branch August 15, 2018 18:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

SIGPIPE on background thread can cause segfault Caught segfault during siege
3 participants