[LibOS] Rework signal handling and syscall emulation

Change log (most important only):
- unify CPU context structures - now we have only one version -
  `PAL_CONTEXT` - which is shared between LibOS and PALs and it should
  depend only on the host architecture (not OS),
- syscalls emulation changed:
  - dedicated LibOS stack is now used for syscalls emulation,
  - removed one indirection level in syscalls table - now it stores
    `shim_do_*` functions directly,
- signal handling - completely rewritten:
  - all signal queues use proper locking schemes now,
  - signals are handled *only* when returning to the user app from LibOS
    or PAL,
  - nested signals are now possible,
  - the app is allowed to jump out of signal handler with the same
    sematics as on normal Linux,
  - signal altstack is now fully supported,
  - syscall restarting is now supported,
  - doing a backtrace from the signal handler works properly,
- disallow injecting host-level signals, with one exception, see
  `sys.enable_sigterm_injection` manifest option for more details.

Documentation/devel/new-syscall.rst

@@ -3,49 +3,34 @@ Implementing new system call
 
 .. highlight:: c
 
-1. Define interface of system call
-----------------------------------
+1. Define interface of system call and add it to system call table
+------------------------------------------------------------------
 
 For example, assume we are implementing :manpage:`sched_setaffinity(2)`. You
-must find the definition of ``sched_setaffinity`` in
-:file:`shim_syscalls.c`, which will be the following code::
+must add the prototype of the function implementing it to :file:`shim_table.h`::
 
-   SHIM_SYSCALL_RETURN_ENOSYS(sched_setaffinity, 3, long, pid_t, pid, unsigned int,
-                              len, unsigned long*, user_mask_ptr)
-
-Change this line to ``DEFINE_SHIM_SYSCALL(...)`` to name the function that
-implements this system call: ``shim_do_sched_setaffinity`` (this is the naming
-convention, please follow it)::
-
-   DEFINE_SHIM_SYSCALL(sched_setaffinity, 3, shim_do_sched_setaffinity, long, pid_t, pid,
-                       unsigned int, len, unsigned long*, user_mask_ptr)
-
-
-2. Add definitions to system call table
----------------------------------------
+   long shim_do_sched_setaffinity(pid_t pid, unsigned int len, unsigned long* user_mask_ptr);
 
-To implement system call ``sched_setaffinity``, two functions need to be defined
-in :file:`shim_table.h`: ``__shim_sched_setaffinity`` and
-``shim_do_sched_setaffinity``. The first one should already be defined. Add the
-second in respect to the system call you are implementing, with the same
-prototype as defined in :file:`shim_syscalls.c`::
+Note that we use the following naming convetion: ``shim_do_`` followed by
+an actual syscall name. Additionally this function should return ``long``.
+Now you need to add an appropriate entry in the syscalls table in
+:file:`shim_table-$(ARCH).c`::
 
-   long shim_do_sched_setaffinity(pid_t pid, unsigned int len, unsigned long* user_mask_ptr);
+    [__NR_sched_setaffinity] = (shim_fp)shim_do_sched_setaffinity
 
-3. Implement system call
+2. Implement system call
 ------------------------
 
-You can add the function body of ``shim_do_sysinfo`` (or the function name defined
-earlier) in a new source file or any existing source file in
-:file:`LibOS/shim/src/sys`.
+You can add the function body of ``shim_do_sched_setaffinity`` in a new source
+file or any existing source file in :file:`LibOS/shim/src/sys`.
 
 For example, in :file:`LibOS/shim/src/sys/shim_sched.c`::
 
    long shim_do_sched_setaffinity(pid_t pid, unsigned int len, unsigned long* user_mask_ptr) {
       /* code for implementing the semantics of sched_setaffinity */
    }
 
-4. Add new PAL Calls (optional)
+3. Add new PAL Calls (optional)
 -------------------------------
 
 The concept of Graphene library OS is to keep the PAL interface as simple as
@@ -66,14 +51,14 @@ Make sure you use the PAL-specific data types, including :type:`PAL_BOL`,
 with the ``Dk`` prefix, followed by a comprehensive name describing the purpose
 of the PAL call.
 
-5. Export new PAL calls from PAL binaries (optional)
+4. Export new PAL calls from PAL binaries (optional)
 ----------------------------------------------------
 
 For each directory in :file:`PAL/host/`, there is a :file:`pal.map` file. This
 file lists all the symbols accessible to the library OS. The new PAL call needs
 to be listed here in order to be used by your system call implementation.
 
-6. Implement new PAL calls (optional)
+5. Implement new PAL calls (optional)
 -------------------------------------
 
 .. todo::

Documentation/libos/shim-init.rst

@@ -7,5 +7,5 @@ LibOS documentation
 
 There is a |~| random function:
 
-.. doxygenfunction:: handle_signals
+.. doxygenfunction:: object_wait_with_retry
    :project: libos

Documentation/manifest-syntax.rst

@@ -231,6 +231,20 @@ This specifies whether to allow system calls `eventfd()` and `eventfd2()`. Since
 eventfd emulation currently relies on the host, these system calls are
 disallowed by default due to security concerns.
 
+External SIGTERM injection
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+::
+
+    sys.enable_sigterm_injection = [1|0]
+    (Default: 0)
+
+This specifies whether to allow for a one-time injection of `SIGTERM` signal
+into Graphene. Could be useful to handle graceful shutdown.
+Be careful! In SGX environment, the untrusted host could inject that signal in
+an arbitrary moment. Examine what your application's `SIGTERM` handler does and
+whether it poses any security threat.
+
 Root FS mount point
 ^^^^^^^^^^^^^^^^^^^
 

Documentation/pal/host-abi.rst

@@ -301,7 +301,7 @@ Exception handling
 
 .. doxygentypedef:: PAL_CONTEXT
    :project: pal
-.. doxygenstruct:: PAL_CONTEXT_
+.. doxygenstruct:: PAL_CONTEXT
    :project: pal
    :members:
 

Examples/apache/httpd.manifest.template

@@ -14,6 +14,9 @@ loader.log_level = "$(GRAPHENE_LOG_LEVEL)"
 # Environment variables
 loader.env.LD_LIBRARY_PATH = "/lib:$(ARCH_LIBDIR):/usr/lib:/usr/$(ARCH_LIBDIR)"
 
+# Allow for injecting SIGTERM signal from the host.
+sys.enable_sigterm_injection = 1
+
 # Mounted FSes. The following "chroot" FSes mount a part of the host FS into the
 # guest. Other parts of the host FS will not be available in the guest.
 

Examples/lighttpd/lighttpd.manifest.template

@@ -20,6 +20,9 @@ loader.insecure__use_cmdline_argv = 1
 # Environment variables for lighttpd
 loader.env.LD_LIBRARY_PATH = "/lib:$(ARCH_LIBDIR):$(INSTALL_DIR)/lib"
 
+# Allow for injecting SIGTERM signal from the host.
+sys.enable_sigterm_injection = 1
+
 # Mounted FSes. The following "chroot" FSes mount a part of the host FS into the
 # guest. Other parts of the host FS will not be available in the guest.
 

Examples/memcached/memcached.manifest.template

@@ -53,6 +53,11 @@ loader.env.LD_LIBRARY_PATH = "/lib:$(ARCH_LIBDIR):/usr/$(ARCH_LIBDIR)"
 # All other environment variables are unset. This is the default Graphene
 # behavior. Memcached doesn't require any environment variables to be set.
 
+################################## SIGNALS ####################################
+
+# Allow for injecting SIGTERM signal from the host.
+sys.enable_sigterm_injection = 1
+
 ################################# MOUNT FS  ###################################
 
 # General notes:

Examples/nginx/nginx.manifest.template

@@ -16,6 +16,9 @@ loader.insecure__use_cmdline_argv = 1
 # Environment variables
 loader.env.LD_LIBRARY_PATH = "/lib:$(ARCH_LIBDIR):/usr/local/lib:/usr/lib:/usr/$(ARCH_LIBDIR)"
 
+# Allow for injecting SIGTERM signal from the host.
+sys.enable_sigterm_injection = 1
+
 # Mounted FSes. The following "chroot" FSes mount a part of the host FS into the
 # guest. Other parts of the host FS will not be available in the guest.
 

Examples/nodejs-express-server/nodejs.manifest.template

@@ -19,6 +19,9 @@ loader.insecure__use_cmdline_argv = 1
 # fs.mount.xxx.uri).
 loader.env.LD_LIBRARY_PATH = "/lib:$(ARCH_LIBDIR):/usr/$(ARCH_LIBDIR)"
 
+# Allow for injecting SIGTERM signal from the host.
+sys.enable_sigterm_injection = 1
+
 # Mount host-OS directory to required libraries (in 'uri') into in-Graphene visible directory /lib
 # (in 'path').
 fs.mount.lib.type = "chroot"

Examples/python-simple/python.manifest.template

@@ -21,6 +21,9 @@ loader.env.PYTHONHOME = "$(PYTHONHOME)"
 loader.env.PYTHONPATH = "$(PYTHONHOME):$(PYTHONHOME)/plat-$(ARCH_LONG):$(PYTHONDISTHOME):$(PYTHONHOME)/lib-dynload"
 loader.env.HOME = "/home/user"
 
+# Allow for injecting SIGTERM signal from the host.
+sys.enable_sigterm_injection = 1
+
 # Mounted FSes. The following "chroot" FSes mount a part of the host FS into the
 # guest. Other parts of the host FS will not be available in the guest.
 

Examples/r/R.manifest.template

@@ -13,6 +13,9 @@ loader.log_level = "$(GRAPHENE_LOG_LEVEL)"
 # Read application arguments directly from the command line. Don't use this on production!
 loader.insecure__use_cmdline_argv = 1
 
+# Allow for injecting SIGTERM signal from the host.
+sys.enable_sigterm_injection = 1
+
 # Environment variables for R
 loader.env.LD_LIBRARY_PATH = "$(R_HOME)/lib:/lib:$(ARCH_LIBDIR):/usr/lib:/usr/$(ARCH_LIBDIR)"
 loader.env.PATH = "$(R_HOME)/bin:/usr/bin:/bin"

Examples/ra-tls-mbedtls/server.manifest.template

@@ -11,6 +11,9 @@ loader.env.LD_LIBRARY_PATH = "/lib:/lib/x86_64-linux-gnu"
 # Read application arguments directly from the command line. Don't use this on production!
 loader.insecure__use_cmdline_argv = 1
 
+# Allow for injecting SIGTERM signal from the host.
+sys.enable_sigterm_injection = 1
+
 # Request remote attestation functionality from Graphene
 sgx.remote_attestation = 1
 

Examples/redis/redis-server.manifest.template

@@ -33,6 +33,11 @@ loader.insecure__use_cmdline_argv = 1
 # - $(ARCH_LIBDIR) is searched for Name Service Switch (NSS) libraries
 loader.env.LD_LIBRARY_PATH = "/lib:$(ARCH_LIBDIR):/usr/$(ARCH_LIBDIR)"
 
+################################## SIGNALS ####################################
+
+# Allow for injecting SIGTERM signal from the host.
+sys.enable_sigterm_injection = 1
+
 ################################# MOUNT FS  ###################################
 
 # General notes:

LibOS/gcc-patches/libgomp-replace-futex-instruction.patch

@@ -1,13 +1,20 @@
 diff --git a/libgomp/config/linux/x86/futex.h b/libgomp/config/linux/x86/futex.h
-index ead74d1496736a49694ef6b9b2b4da50f9852664..3c82859ad8b82a09ec95f720727937ee5a2863c1 100644
+index ead74d1496736a49694ef6b9b2b4da50f9852664..917412326135b9e9da3e14da3765868d217f1ed3 100644
 --- a/libgomp/config/linux/x86/futex.h
 +++ b/libgomp/config/linux/x86/futex.h
-@@ -30,13 +30,16 @@
+@@ -30,13 +30,23 @@
  #  define SYS_futex	202
  # endif
 
-+asm (".weak syscalldb\r\n"
-+     ".type syscalldb, @function\r\n");
++asm(
++".weak syscalldb\n"
++".type syscalldb, @function\n"
++".macro SYSCALLDB\n"
++"leaq .Lafter_syscalldb\\@(%rip), %rcx\n"
++"jmpq *syscalldb@GOTPCREL(%rip)\n"
++".Lafter_syscalldb\\@:\n"
++".endm\n"
++);
 +
  static inline void
  futex_wait (int *addr, int val)
@@ -16,34 +23,34 @@ index ead74d1496736a49694ef6b9b2b4da50f9852664..3c82859ad8b82a09ec95f720727937ee
 
    register long r10 __asm__("%r10") = 0;
 -  __asm volatile ("syscall"
-+  __asm volatile ("subq $128, %%rsp; callq *syscalldb@GOTPCREL(%%rip); addq $128, %%rsp;"
++  __asm volatile ("SYSCALLDB"
  		  : "=a" (res)
  		  : "0" (SYS_futex), "D" (addr), "S" (gomp_futex_wait),
  		    "d" (val), "r" (r10)
-@@ -45,7 +48,7 @@ futex_wait (int *addr, int val)
+@@ -45,7 +55,7 @@ futex_wait (int *addr, int val)
      {
        gomp_futex_wait &= ~FUTEX_PRIVATE_FLAG;
        gomp_futex_wake &= ~FUTEX_PRIVATE_FLAG;
 -      __asm volatile ("syscall"
-+      __asm volatile ("subq $128, %%rsp; callq *syscalldb@GOTPCREL(%%rip); addq $128, %%rsp;"
++      __asm volatile ("SYSCALLDB"
  		      : "=a" (res)
  		      : "0" (SYS_futex), "D" (addr), "S" (gomp_futex_wait),
  			"d" (val), "r" (r10)
-@@ -58,7 +61,7 @@ futex_wake (int *addr, int count)
+@@ -58,7 +68,7 @@ futex_wake (int *addr, int count)
  {
    long res;
 
 -  __asm volatile ("syscall"
-+  __asm volatile ("subq $128, %%rsp; callq *syscalldb@GOTPCREL(%%rip); addq $128, %%rsp;"
++  __asm volatile ("SYSCALLDB"
  		  : "=a" (res)
  		  : "0" (SYS_futex), "D" (addr), "S" (gomp_futex_wake),
  		    "d" (count)
-@@ -67,7 +70,7 @@ futex_wake (int *addr, int count)
+@@ -67,7 +77,7 @@ futex_wake (int *addr, int count)
      {
        gomp_futex_wait &= ~FUTEX_PRIVATE_FLAG;
        gomp_futex_wake &= ~FUTEX_PRIVATE_FLAG;
 -      __asm volatile ("syscall"
-+      __asm volatile ("subq $128, %%rsp; callq *syscalldb@GOTPCREL(%%rip); addq $128, %%rsp;"
++      __asm volatile ("SYSCALLDB"
  		      : "=a" (res)
  		      : "0" (SYS_futex), "D" (addr), "S" (gomp_futex_wake),
  			"d" (count)

LibOS/glibc-patches/glibc-2.23.patch

@@ -1,5 +1,5 @@
 diff --git a/Makeconfig b/Makeconfig
-index 87a22e88bed375d073663063dd4a93444d72ba25..b69ff41077a5f279bde52aac11f03604fa61ae65 100644
+index 87a22e88bed375d073663063dd4a93444d72ba25..3e2cf93fad804b81e67a476398a9c86c93ffa3cd 100644
 --- a/Makeconfig
 +++ b/Makeconfig
 @@ -841,7 +841,8 @@ endif	# $(+cflags) == ""
@@ -482,15 +482,15 @@ index 56e085685804f51e49eef4c21ad9efdf0d21e59e..81038f485616592419c5fb2f63c0c887
  	cfi_adjust_cfa_offset(-8)
  	cmpq	$-4095, %rax		/* Check %rax for error.  */
 diff --git a/sysdeps/unix/sysv/linux/x86_64/sigaction.c b/sysdeps/unix/sysv/linux/x86_64/sigaction.c
-index 71ac05c4bc01c560935f3bbd4306e3aeacb6d9be..c6e28bb2362b07cfa3f3ade189ae337195671830 100644
+index 71ac05c4bc01c560935f3bbd4306e3aeacb6d9be..2f5e622a88b40f5f175455fe64b254b70b36ff25 100644
 --- a/sysdeps/unix/sysv/linux/x86_64/sigaction.c
 +++ b/sysdeps/unix/sysv/linux/x86_64/sigaction.c
 @@ -120,7 +120,7 @@ asm									\
     "	.type __" #name ",@function\n"					\
     "__" #name ":\n"							\
     "	movq $" #syscall ", %rax\n"					\
 -   "	syscall\n"							\
-+   SYSCALLDB_ASM							\
++   SYSCALLDB								\
     ".LEND_" #name ":\n"							\
     ".section .eh_frame,\"a\",@progbits\n"				\
     ".LSTARTFRAME_" #name ":\n"						\

LibOS/glibc-patches/glibc-2.27.patch

@@ -1,5 +1,5 @@
 diff --git a/Makeconfig b/Makeconfig
-index 86a71e580213f6e5de6e619d9f3370f4365b4ae2..114d4ee080dbccbb696f40469a8010663fec7a9a 100644
+index 86a71e580213f6e5de6e619d9f3370f4365b4ae2..515f8be0d13a14fe0588c357cfc5608503aca341 100644
 --- a/Makeconfig
 +++ b/Makeconfig
 @@ -916,7 +916,8 @@ endif	# $(+cflags) == ""
@@ -290,15 +290,15 @@ index 4a9b662074ddbd62d958d8db7ec09b9d5741ad59..21bd14d778135102632952712a307453
  	cfi_adjust_cfa_offset(-8)
  	cmpq	$-4095, %rax		/* Check %rax for error.  */
 diff --git a/sysdeps/unix/sysv/linux/x86_64/sigaction.c b/sysdeps/unix/sysv/linux/x86_64/sigaction.c
-index 2f7459f6fc602d1a176f725bb2dd4f10364ae997..9c2191541be0b8855ee3789993004a03bf9eab0f 100644
+index 2f7459f6fc602d1a176f725bb2dd4f10364ae997..b4c35a8e494ce1a8a9faa6ffd517dc993465b8be 100644
 --- a/sysdeps/unix/sysv/linux/x86_64/sigaction.c
 +++ b/sysdeps/unix/sysv/linux/x86_64/sigaction.c
 @@ -120,7 +120,7 @@ asm									\
     "	.type __" #name ",@function\n"					\
     "__" #name ":\n"							\
     "	movq $" #syscall ", %rax\n"					\
 -   "	syscall\n"							\
-+   SYSCALLDB_ASM							\
++   SYSCALLDB								\
     ".LEND_" #name ":\n"							\
     ".section .eh_frame,\"a\",@progbits\n"				\
     ".LSTARTFRAME_" #name ":\n"						\

LibOS/glibc-patches/glibc-2.31.patch

@@ -1,5 +1,5 @@
 diff --git a/Makeconfig b/Makeconfig
-index f252842979a1d777e0f0c2bdafa7a65aee0805cd..4e22deb595ad6e5dd36571aec01d5b1e39f10f16 100644
+index f252842979a1d777e0f0c2bdafa7a65aee0805cd..0a044a7b60ae305af56c47982e3bdf1e25769527 100644
 --- a/Makeconfig
 +++ b/Makeconfig
 @@ -930,7 +930,8 @@ endif	# $(+cflags) == ""
@@ -246,15 +246,15 @@ index 31bbc9dbe4bec944bb0037d24900120535d08504..e1bfe2f13eb1a6f00e461973e09ef426
  	   leaving RDI and RSI available for use later can avoid
  	   shuffling values.  */
 diff --git a/sysdeps/unix/sysv/linux/x86_64/sigaction.c b/sysdeps/unix/sysv/linux/x86_64/sigaction.c
-index c58a77c5c6a3547a3bff5437c1a2b13368f2f48b..f1b413ea733966eb3e649bb30a49ffc3c65d2fa9 100644
+index c58a77c5c6a3547a3bff5437c1a2b13368f2f48b..fdaef663d6f9a69901a612970485d3673eb65e95 100644
 --- a/sysdeps/unix/sysv/linux/x86_64/sigaction.c
 +++ b/sysdeps/unix/sysv/linux/x86_64/sigaction.c
 @@ -78,7 +78,7 @@ asm									\
     "	.type __" #name ",@function\n"					\
     "__" #name ":\n"							\
     "	movq $" #syscall ", %rax\n"					\
 -   "	syscall\n"							\
-+   SYSCALLDB_ASM							\
++   SYSCALLDB								\
     ".LEND_" #name ":\n"							\
     ".section .eh_frame,\"a\",@progbits\n"				\
     ".LSTARTFRAME_" #name ":\n"						\
@@ -391,15 +391,15 @@ index 776d2fc610751b7efdcf5b39aa42728504b14080..9fc94470d650f3c80e9c0a4ec04c8754
  #if !SHSTK_ENABLED
  	/* Push back the return PC.  */
 diff --git a/sysdeps/unix/sysv/linux/x86_64/x32/times.c b/sysdeps/unix/sysv/linux/x86_64/x32/times.c
-index fb93cb609c09831cbcee1c0cc57ff7a6bbe12ff5..c36fd5264edc96398e49ac98bcf213b7f761f4d5 100644
+index fb93cb609c09831cbcee1c0cc57ff7a6bbe12ff5..380f0bc4ede8249215c423430d43566a011943c9 100644
 --- a/sysdeps/unix/sysv/linux/x86_64/x32/times.c
 +++ b/sysdeps/unix/sysv/linux/x86_64/x32/times.c
 @@ -26,7 +26,7 @@
      TYPEFY (arg1, __arg1) = ARGIFY (arg1);			 	\
      register TYPEFY (arg1, _a1) asm ("rdi") = __arg1;			\
      asm volatile (							\
 -    "syscall\n\t"							\
-+    SYSCALLDB_ASM                           \
++    SYSCALLDB								\
      : "=a" (resultvar)							\
      : "0" (number), "r" (_a1)						\
      : "memory", REGISTERS_CLOBBERED_BY_SYSCALL);			\