linux_101.txt

## Apropos:

# LINUX ADMINISTRATION RECIPES

## External Links: [[{]]
* <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/>
* <http://www.linuxfromscratch.org/>
* <https://linux.die.net/man/>
* <https://linux.die.net/Linux-CLI/>
* <https://en.wikipedia.org/wiki/Linux_Standard_Base>
* <https://docs.fedoraproject.org/en-US/fedora/f34/system-administrators-guide/>
* <https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/>
[[}]]

# Linux Basics [[{linux.101]]
Linux itself is just an OS kernel in charge of sharing the LIMITED hardware
resources amongst potentially many running tasks and users working simultaneously
on the systems. More preciselly, the main tasks in charge of kernel control are:
* Orchestate(schedule) how much time each running-task is allowed to
  run on each CPU before such task is put on stand-by to let
  another task proffit from such CPU.
* Assign which process has access to the (limited)
  RAM memory on the system and move data on RAM used by stand-by processes
  to secondory storage (disks) in case of RAM space shortage.
* Provide support for users and permissions, so that different users will
  be able to isolate and protect its data from other users.

  Kernel control is transparent to running tasks or processes, so user-space task
will run with the perception that they run alone in its own CPU and with
all available memory for themself. When the kernel puts them on-hold
such tasks will be freezed and once restarted it will NOT notice any
change to the state before being put on-hold. Only the kernel is aware of
the complex trickeries needed to make tasks run in parallel and isolated
from each other.<br>

Other important tasks offered by the kernel are:
* Abstract the running hardware into standarized interfaces, so
  user application will not need to work differently with different hardware.
  For example an application will ask the kernel to write data to the disk and
  the kernel will take care of the internal difference between the miriads
  of diferent disk hardware technologies.
* Provide an easy-to-use file-system to work with data on disk, so that apps
  can orginized data in terms of files and directories, instead of just
  bunch-of-bytes on the hard-disk.
* Provide network communication and support for standard network protocols
  like TCP/UP, bluetooth, WiFI, ... so that each app does not need to reimplement
  them.
* Provide mechanisms to allow two running tasks to communicate with each other
  at will.

## Kernel mode vs User mode
* When the CPU is executing kernel code it's running with elevated privileges.
  The software has full control of the hardware and can do "anything" on
  the system and access all RAM, disk, network resources at will.
* Standard applications will run in user-mode and will have limited access
  to only the RAM memory assigned by the kernel. They will not be able to
  inspect the memory of other processes running on the system. In fact they
  are not aware of such memory due to the many trickeries done by the kernel
  to isolate each task.

## Files, files and more files

  Any running process needs some incomming data to work with and
produced new data that must be stored somewhere.

   This data can be provided by some storage system (hard-disk,
  usb, tape, ...), arrive (continuosly) from the network,
  or be generated by another concurrent process.

   Linux (UNIX actually) treats all input data sources and
  output data sinks as *"file devices"*.

  Internally there can be many differences (block devices with
random access vs char devices with just sequential access), but
running processes mostly always use the file methaphor to access
all of them.

   Any running process will have 3 devices available "for free":
* STDIN : The standard input  file.
* STDOUT: The standard output file
* STDERR: The standard error  file

   The standard shell provides many utilities to juggle with those
three standard files. In particular it allows to easely forward
the STDOUT output from a running-process to the STDIN input of
another running process using the "|" pipe syntax:
```
| $ command1  | command2  #
|             ^
|             └─ Send STDOUT output from command1 to
|                 STDIN   input   of command2
```

  STDOUT and STDERR by default are assigned to the running-process
associated terminal (the console where the user has been logged).
The shell allows also to redirect STDOUT/STDERR to any other
file in our file system. Ex:
  ```
  | $ command1 1>output.log         2>error.log
  │            └─────┬────┘         └─────┬───┘
  |            redirects STDOUT(1)  redirects STDERR
  |            to output.log        to error.log
  |
  | $ command1 1>output.log         2>&1
  |            └─────┬────┘         └─┬┘
  |            redirects STDOUT(1)  redirects STDERR
  |            to output.log        to STDOUT (&1, aka output.log)
  ```
[[}]]

# Process model [[{linux.101.process_model,job_control.101,]]

  Linux follows a parent-child process model.

  Once loaded and initialized during the boot process,
the kernel will launch an initial user-space process in
charge of reading system configuration and (re)start
all other user-space processes that builds a running system.

 Normally this initial process is systemd in modern
systems (or initd in older or embedded ones).

 Each process can optionally launch new children processes
up to the resource limits established on the running system.

 By default a child-process inherits the same user (and so, permissions)
than its parent process. Some processes like the remote
login "sshd" service (ssh is an acronymn for secure-shell) will
change the child-process user/permission to a less privileged
account.

  A simplified process-tree of a running-system will look like:      [[{doc_has.diagram]]

  ```
  | PROCESS                                       USER    PROCESS   PARENT-ID
  |                                                      UNIQUE-ID
  | systemd·······································root       1         0
  |       └─crond·································root      23         1
  |       |-cupsd·································root      70         1
  |       |-rtkit-daemon··························rtkit    100         1
  |       |-sshd··································root      10         1
  |       |    └─sshd·····························mike     300        10
  |       |         └─bash························mike     301       300
  |       |              └─firefox················mike     302       301
  |       |-systemd·······························alice    705         1
  |       |       └─at-spi-bus-laun···············alice    706       705
  |       |       |···············└─dbus-daemon···alice    707       706
  |       |       |-gnome-terminal················alice    883       705
  |       |                       ─bash-+·········alice    884       883
  |       |                             └─top·····alice    885       884
  |       |-systemd-journal·······················root      10         1
  |       ...
  | Notice for example that the same process "bash" runs as a user or another
  | ( *mike* or  *alice*) depending on the "path"
  | followed until the process is executed.                           [[}]]
  ```

* The initial sshd running as root user, will span a new sshd child process
  with restricted  *"mike"* privileges/permissions once the user
  has introduced its correct user and password in the remote ssh session, and
  from there on, all children will just be able to run with  *"mike"*
  privileges/permissions.
* Similarly the root systemd process will span a new child process will
  restricted  *"alice"* privileges/permissions once logged in
  the local console, and from there on, all children will just be able to
  run with  *"alice"* privileges/permissions.


## Executable file vs in-memory process [[{linux.101]]

 Applications are stored on disk drives as files or
"bunch-of-instructions and initial data".

 When the kernel executes and application it will read the executable file, load
the "bunch-of-instructions" into RAM memory, setup the initial data, assign
restricted privileges and finally allow the program-in-memory to be executed by
any free-available CPU on the system.
[[}]]


## Basic file permissions  [[{linux.101,security.aaa]]
Standar file permissions allows to assign different access permissions to
the owner of the file, the group owner of the file and anyone else.

  ```
  | $ ls -l myFileOfInterest.db
  |
  | -rw?-r-?---? john accountDepartment ....  myFileOfInterest.db
  | └┼┘│└┼┘│└┼┘│  └┬─┘ └──────┬────────┘
  |  │ │ │ │ │ │   │          │
  |  │ │ │ │ │ │   │          └─ group owner
  |  │ │ │ │ │ │   └──────────── user  owner
  |  │ │ │ │ │ │
  |  │ │ │ │ │ └──────────────── sticky bit (hidden if not set)
  |  │ │ │ │ │
  |  │ │ │ │ └────────────────── permissions allowed to others: read           access
  |  │ │ │ │
  |  │ │ │ └──────────────────── SUID bit   (hidden if not set)
  |  │ │ │
  |  │ │ └────────────────────── permissions allowed to group : read           access
  |  │ │
  |  │ └──────────────────────── SUID bit   (hidden if not set)
  |  │
  |  └────────────────────────── permissions allowed to user  : read&amp;write access
  ```

The previous command line can be read as:

> Allow read and write permissions to file-owner "john",
> read permissions to group-owner "accountDepartment"
>  and no   permissions to anyone-else """

  ```
  |            ┌──────┬─────────────────────────────┬─────────────────────────────┐
  |Permissions │Symbol│      FILE                   │     DIRECTORY               │
  |┌───────────┼──────┼─────────────────────────────┼─────────────────────────────┤
  |│       read│  r   │ Allows to read the content  │ Allows to list the files in │
  |│           │      │ of the file                 │ and file-attributes in the  │
  |│           │      │                             │ directory                   │
  |├───────────┼──────┼─────────────────────────────┼─────────────────────────────┤
  |│      write│  w   │ Allows to write, modify,    │ Allows to add and delete    │
  |│           │      │ append or delete the file   │ files into the directory and│
  |│           │      │ content.                    │ modify metadata (access     │
  |│           │      │                             │ or modification time, ...)  │
  |├───────────┼──────┼─────────────────────────────┼─────────────────────────────┤
  |│    execute│  x   │ Allows to execute the       │ Allows to enter into the    │
  |│           │      │ program or script           │ directory                   │
  |├───────────┼──────┼─────────────────────────────┴─────────────────────────────┤
  |│    sticky │  T   │ Only the person that created the file/dir. can change it, │
  |│           │      │ even if other people have write permissions to file/dir.  │
  |│           │      │ turn on: $ chmod +t someFileOrDir                         │
  |│           │      │ Normal /tmp (temporary user files) is an sticky directory │
  |├───────────┼──────┼───────────────────────────────────────────────────────────┤
  |│       suid│  S   │ Allow SUID/SGID (switch user/group ID). When executed it  │
  |│           │      │ it will be executed with creator/group of file, instead of│
  |│           │      │ current user.                                             │
  |│           │      │ turn on: $ chmod +s someFileOrDir                         │
  |└───────────┴──────┴───────────────────────────────────────────────────────────┘
  ```
[[}]]

## User Management [[{security.aaa.101,linux.101,]]
  |```
  | $ useradd -D                       ← display (D)efault values to apply when creating new uses.
  |                                       (Default group / HOME, SHELL, ...)
  |                                       Use next flags to update defaults:
  |                                       --base-dir BASE_DIR  :  (default to /home) Ignored if --home-dir set
  |                                       --expiredate EXPIRE_DATE
  |                                       --inactive INACTIVE  :  day # after pass.expiration before disabling
  |                                       --gid GROUP: existing group name or ID for initial group (when --no-user-group used)
  |                                       --shell SHELL
  |
  | $ addgroup PROJECT01               ← Best pattern when allowing multi-user access to the linux machine.
  |                                       Create a new group per project ,then assign permissions to group (vs
  |                                       to user). This greately simplifies permissions administration in
  |                                       multi-user setups.
  |
  | $ useradd alice                 \   ← Create user alice.
  |     --shell=/usr/bin/git-shell  \   ← Assign shell (restricted git-shell in this example)
  |     --shell=/usr/bin/git-shell  \   ← upon succesful login alice will be presented with a git shell *1
  |     --groups PROJECT01,PROJECT02\   ← Assign PROJECT01 as default group for alice.
  |     --password ...              \   ← create an initial and strong password (or provide alice with ssh keys.
  |     --no-user-group             \   ← Do NOT create a new group (otherwise indicated by --gid)
  |     --no-create-home            \   ← Do not create the /home/alice directory. (Some sort of
  |                                       /var/lib/git/PROJECT01 directories already exist with read/write permissions
  |                                       for PROJECT01, ... groups)
  |
  |                                     *1 By default /bin/bash o similar is used. git-shell restricts
  |                                        access to just git-related commands.
  |  Other useful 'useradd' options are:
  |  --skel SKEL_DIR :  skel. dir. to be copied in the user's home directory
  |  --key KEY=VALUE : Overrides /etc/login.defs defaults
  |                    (UID_MIN, UID_MAX, UMASK, PASS_MAX_DAYS and others).
  |                    Example: -K PASS_MAX_DAYS=-1 can be used when creating
  |                      system account to turn off password ageing, even though
  |                      system account has no password at all.
  |  --no-log-init   : Do not add user to lastlog and faillog databases
  |  --non-unique    : Allow duplicate (non-unique) existing UID in --uid. Most of the times used to
  |                    allow 'root' users with non-root name accounts.
  |  --system        : Create system account (no aging, uid chosen in SYS_UID_MIN-SYS_UID_MAX range)
  |  --root CHROOTDIR: Apply changes in chrooted directory.
  |  --uid UID       : numerical value for user's ID.
  |  --user-group    : Create group with the same name as user, and use as initial group
  |  --selinux-user SEUSER : SELinux user for the user's login
  ```


## su/sudo: Switch user:  [[{security.aaa,linux.101,PM.WiP]]

* su and sudo are mostly used to allow temporal root/superuser access
  to standard users for administrative tasks like installing new applications,
  re-configuring the network, ...
* sudo is ussually considered safer than su. Ubuntu was the first
  distribution to allow sudo-only. Others distributions are also
  changing to sudo-only as time passes.
* sudo offers also a plugable architecture not offered by su
  to provide different authentication and audit mechanisms.
  REF:
  * https://www.sudo.ws/ (Sudo Home page)
  * https://www.sudo.ws/plugins.html (sudo Third-party plugins)
    https://linux.die.net/man/1/su
    https://linux.die.net/man/8/sudo
* Ex. ussage:
  ```
  | $ sudo vim /etc/passwd # edit /etc/passwd as root
  | $ su  # Change to root user
  ```
[[}]]


[[security.aaa.101}]]

[[{101,job_control.task_scheduling,PM.TODO]]
## Job/Process control: Scheduling Tasks
Officially in Linux/UNIX/Posix, a Job or task is a running process.

* cron    : program task to be run periodically
* at
* anacron :
  Unlike cron(8), it does not assume that the machine is running continuously.
  Hence,  it  can  be used on machines that aren't running 24 hours a day.

  Anacron checks whether this job has been executed in the last n days.
  If not, Anacron runs the job's shell command, after waiting for the
  number of minutes specified as the delay parameter.
  NOTE/WARN: Only the date , not hour is used.

  e.g.: On Debian-based systems, anacron will be activated hourly every day
        from 07:30  local  time  to 23:30  local time through cron job

[[}]]


[[{101.text_utils,use_case.*,PM.low_code]]
# Text Search and processing

## Text view 

* Displaying text:
  ```
  $ head -n 20 /path/to/textFile # shows first 20 lines (-n 20). 10 lines by default if -n not provided.
  $ tail -n 20 /path/to/textFile # shows last  20 lines (-n 20). 10 lines by default if -n not provided.
  $ tail -f    /path/to/textFile # shows stream ("f"lush) of lines as they are appended to the file
  $ less       /path/to/textFile # Views text. Add scroll control backwards and forwards.
                                 # embedded systems use "more", that just scroll forwards.
  $ cat        /path/to/textFile # dump text content to standard output (STDOUT)
  $ cat file1 file2 file3 ....   # concatenates files'content and dumps into STDOUT
  $ tac file1 file2 file3 ....   # concatenates files'content and dumps into STDOUT in reverse order
  ```

  ```
  | man 1 column:
  |$ column -n 140 /usr/share/dict/words  # ← One token per line input
  |→ A             archdeacon      effort's      mads             salient's
  |→ A's           archdeacon's    effortless    madwoman         salients
  |→ ...
  |→ archbishop    effigy          madras        salesperson's    étude's
  |→ archbishop's  effigy's        madras's      salespersons     études
  |→ archbishopric effluent        madrases      saleswoman
  |
  |$ COLUMNS=100 column -t -s, input.csv # ← -t(able):
  |1     2     3                             -s(eparator): Indicates column sep
  |a     b     c                                          (white space by default)
  |x     y     z
  ```
[[}]]

[[{]]
## Extract text info/statistics/difference

  ```
  | $ wc /path/to/textFile  ← Display total count of words, lines and bytes
  |                           Options:
  |                             -w count only words
  |                             -l count only lines
  |                             -w count only bytes
  | 
  | $ diff file1 file2     ← Compares two text files and output a difference report indicating:
  |                          '> line_in_file2_not_in_file1
  |                          '< line_in_file1_not_in_file2
  | $ sdiff                ← Similar to diff but with report in colum mode (more human readable)
  | $ diff3                ← diff for three files
  ```
[[}]]

## SORTING FILE CONTENT !!!! [[{]]
  ```
  | $ sort file1       <-·· Sorting alphabetically lines in file
  |                        (-r to reverse, -g for numerical sort)
  |
  | $ sort file1 \
  |   -t ':'           <-·· Use ':' as separator,
  |   -k 4 -k 1             sort first by column 4, then column 1.
  |
  | $ uniq file1       <··· Eliminates duplicate entries from a file
  |                         Commonly used with sort like:
  |                         $ cat file.txt | sort | uniq
  |                         Options: -c: display number of occurances of each duplicate
  |                                  -u: list only unique entries
  |                                  -d: list only duplicate entries
  |
  | $ join file1 file2 <-·· Join two lines together assuming they share
  |                         at least one common value on the relevant line,
  |                         skiping lines withouth common value.
  ```
[[}]]

## Text edit  [[{]]

* By default we can use the following tools like:
  ```
  | $ tool inputFile   # Apply to given file
  |  or
  | $ command1 ... | tool1 | tool2 | tool3 # Pipe execution output (stdout) as input
  |                                          for next command.
  |
  |                                 CUT CONTENT BY COLUMN:
  | $ cut -d "," -f 1,3,7 file1.csv #<· Use "," as column delimiter,
  |                                 #   show then columns 1,3,7
  | $ cut -c 1-50         file.txt  #<· show characters 1 to 50 in each line
  | $ cut -5, 8, 20-      file.txt  #<· show characters 1 to 5 and 8 and
  |                                    from 20 to the end
  |    
  | $ tr "u" "d" file1              # <· replace characters in file
  | $ cat some_file | \
  |   tr '[A-Z]' '[a-z]' > new_file # <· Ex: Convert all capital letters to lowercase
  |
  | $ nl file1.txt                  # <· Display file1.txt to STDOUT prefixing with line numbers
  | 
  | $ sed "s/ *up/ down/g" file1.txt <- Replaces " *up*" by " *down*".
  |                                     /g flag: replace all ocurrences (vs only first match).
  | NOTE: sed stands for (s)tream (ed)itor. It has lot of powerful flags 
  |       for instance, searching for regular expresions ("complex matches).
  ```
[[}]]

[[101.text_utils}]]




# Monitoring

## basic process monit/control [[{job_control.101,linux.101,monitoring.jobs,monitoring.i/o]]

* shows list of the processes running:
  ```
  | $ ps  -a  # Without options: display processes belonging to current
  |           # user and with a controlling terminal
  |           # -a : flag to all processes from all users
  |           # Other common "flags" include:
  |           # -u: add user names, %cpu usage, and %mem usage,...
  |           # -x: add also processes without controlling terminals
  |           # -l: add information including UID and nice value
  |           # --forest: show process hierarchy.
  |
  | $ pstree # <·· show parent/children process tree (-p flag show pid)
  |
  | $ top -n # <·· Display "top processes" (by cpu ussage) and finish
  |
  | $ top    # <·· real-time display processes ordered by memory/CPU/...(as in CPU usage)
  |
  |   Z,B,E,e   Global: 'Z' colors; 'B' bold; 'E'/'e' summary/task memory scale
  |   l,t,m     Toggle Summary: 'l' load avg; 't' task/cpu stats; 'm' memory info
  |   0,1,2,3,I Toggle: '0' zeros; '1/2/3' cpus or numa node views; 'I' Irix mode
  |   f,F,X     Fields: 'f'/'F' add/remove/order/sort; 'X' increase fixed-width
  |
  |   L,&,<,> . Locate: 'L'/'&' find/again; Move sort column: '<'/'>' left/right
  |   R,H,V,J . Toggle: 'R' Sort; 'H' Threads; 'V' Forest view; 'J' Num justify
  |   c,i,S,j . Toggle: 'c' Cmd name/line; 'i' Idle; 'S' Time; 'j' Str justify
  |   x,y     . Toggle highlights: 'x' sort field; 'y' running tasks
  |   z,b     . Toggle: 'z' color/mono; 'b' bold/reverse (only if 'x' or 'y')
  |   u,U,o,O . Filter by: 'u'/'U' effective/any user; 'o'/'O' other criteria
  |   n,#,^O  . Set: 'n'/'#' max tasks displayed; Show: Ctrl+'O' other filter(s)
  |   C,...   . Toggle scroll coordinates msg for: up,down,left,right,home,end
  |
  |   k,r       Manipulate tasks: 'k' kill; 'r' renice
  |   d or s    Set update interval
  |   W,Y       Write configuration file 'W'; Inspect other output 'Y'
  |   q         Quit
  |
  | $ iotop    # ← Simple top-like I/O monitor
  | <https://linux.die.net/man/1/iotop>
  ```

  ```
  $  kill - # <·· Display existing signals (Default to SIGTERM that most of the times
            #     will just terminate the process "cleanely")
  >  1) SIGHUP   2) SIGINT   3) SIGQUIT  4) SIGILL   5) SIGTRAP
  >  6) SIGABRT  7) SIGBUS   8) SIGFPE   9) SIGKILL 10) SIGUSR1
  > 11) SIGSEGV 12) SIGUSR2 13) SIGPIPE 14) SIGALRM 15) SIGTERM
  > 16) SIGSTKFLT   17) SIGCHLD 18) SIGCONT 19) SIGSTOP 20) SIGTSTP
  > 21) SIGTTIN 22) SIGTTOU 23) SIGURG  24) SIGXCPU 25) SIGXFSZ
  > 26) SIGVTALRM   27) SIGPROF 28) SIGWINCH    29) SIGIO   30) SIGPWR
  > 31) SIGSYS  34) SIGRTMIN    35) SIGRTMIN+1  36) SIGRTMIN+2  37) SIGRTMIN+3
  > 38) SIGRTMIN+4  39) SIGRTMIN+5  40) SIGRTMIN+6  41) SIGRTMIN+7  42) SIGRTMIN+8
  > 43) SIGRTMIN+9  44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12 47) SIGRTMIN+13
  > 48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14 51) SIGRTMAX-13 52) SIGRTMAX-12
  > 53) SIGRTMAX-11 54) SIGRTMAX-10 55) SIGRTMAX-9  56) SIGRTMAX-8  57) SIGRTMAX-7
  > 58) SIGRTMAX-6  59) SIGRTMAX-5  60) SIGRTMAX-4  61) SIGRTMAX-3  62) SIGRTMAX-2
  > 63) SIGRTMAX-1  64) SIGRTMAX

  $ kill [ -s (signal name)] 'process_id'  # <·· Send signal to process.
                                           #     kill -9 kills unconditionally
  $ killall "process_name"  # <·· send signal to all processes matching full name
  $ pkill "process_name"    # <·· send signal to all processes matching part of the name
  $ skill                   # <·· send a particular signal to command/username/tty
                                  -L --- list the various signals that can be sent
                                  -u --- specify a username;
                                  -p --- process id (followed by the process id)
                                  -c --- command name (this is the same as killall)
                                  -t --- (tty number)
                                  -v --- verbose mode
                                  -i --- interactive mode.
  ```

* Kill example. PAUSE AND CONTINUE A PROCESS.<br/>
  When paused, the process does not consume any resource.
  ```
  $ kill -STOP "pid" # Pauses
  $ kill -CONT "pid" # Continues
  ```

* Change process execution priority
  ```
  | $ nice -20 make    # <·· Sets make priority to -20
  |                    #     -20 is maximum priority   (negative only allowed to root)
  |                    #      20 is the minimum priority.
  | $ renice 10 $PID   # <·· Changes priority of process identified by <$PID>.
  |                    #     Use `ps -a` or `top -n` or ... to fetch the PID value
  ```
[[}]]


## Disk Space [[{troubleshooting.disk_full]]
  ```
  |  $ df -kh             # <·· (D)isk (F)ree report.
  |                       #     -k : take block-size=1K
  |                       #     -h : human-readable, print sizes in powers of 1024 (e.g., 1023M)
  | 
  |  $ du -sch dir1 dir2  # <·· (D)isk (U)ssage report for files&directories.
  |                       #     -s: summarize. Display only a total for each argument
  |                       #     -c: produce grand total
  |                       #     -h : human-readable
  ```
[[}]]

## `pv` Pipe viewer [[{monitoring.pipes,monitoring.i/o,monitoring.101,profiling.storage.FS,storage.profiling]]

* <http://ivarch.com/programs/pv.shtml>
* terminal-based tool for monitoring the progress of data through a pipeline.
  ("visual hint" about data "travelling" through processes / pipes / network) 
* pv provides time elasped, %compl., progressbar, throughput, ETA, ...on STDERR.

Example ussage:
 ```
 | $ sudo apt install pv
 | 
 | $ cat /dev/zero    | pv > /dev/null  # Check CPU/memory performance     [[monitoring.cpu]]
 | 2,01GiB 0:00:15 [ 5.38GiB/s] <=> 
 | $ cat /dev/urandom | pv > /dev/null  # Check urandom    performance
 | 2,01GiB 0:00:15 [  430MiB/s] <=>
 | $ pv file | nc -w 1 somewhere.com 3000  # Check network perf.           [[monitoring.network]]
 | $ pv -EE /dev/sda > disk-image.img      # Check disk image (skip errors)[[monitoring.storage.blocks]]
 ```
[[monitoring.pipes.pv}]]


* man pv summary
 ```
 | OUTPUT MODIFIERS
 |   --wait: until first byte has been transferred.
 |   --delay-start SEC (before showing info)
 |   --size SIZE: Set total size (if piping from STDIN and size is known)
 |   --line-mode: (count lines vs counting bytes)
 |                --null for null terminated.
 |   --interval SEC: (1 sec by default)
 |   --width N / --height N: Assume width / height for terminal
 |   --name NAME: Useful with --cursor for "compilcated pipelines"
 |                --cursor: Use cursor pos. escape sequences (vs CR)
 |   --force: (even if STDOUT is not terminal)
 | DATA TRANSFER MODIFIERS
 |   --rate-limit RATE
 |   --buffer-size BYTES
 |   --no-splice: Never  use splice(2). Ussually more efficient way of
 |                transferring data from/to pipe than regular read/write.
 |                but means that the transfer buffer may not be used.
 |                This prevents -A and -T from working.
 |   -E/--skip-errors (set twice to only report a read error once per file)
 |   --stop-at-size
 |   --watchfd PID[:FD]. Wath FIle Descriptor of process PID, and show its
 |                       progress.
 |   --remote PID : where PID is an running instance of pv already running.
 |                  Add other commands to such instance.
 |                  --pidfile FILE save PID of (first running) pv instance.
 ```
[[}]]

[[{PM.TODO]]
## Netdata (Glances++) 
* <https://github.com/netdata/netdata>
* (Used by AWS, ..)
* Visual tool to monitor the data "moving around" in our system.
[[PM.TODO}]]

## Glances [[{monitoring.101,PM.low_code,QA.UX,troubleshooting]]
* <https://www.tecmint.com/glances-an-advanced-real-time-system-monitoring-tool-for-linux/>
* <https://github.com/tldr-pages/tldr/blob/master/pages/common/glances.md>
* Alternative to the `top` common
* Linux/MacOS/FreeBSD python command-line,  curses-based, using psutils
  under the hood.
* It makes easier to find an application/process consuming lots
  of system resources  by highlighting programs consuming too much
  resources and providing maximum of information about the server.
* Allows to define thresholds (careful, warning and critical) in
  config files.
* Display info about:
  * CPU (user, kernel, idle processes).
  * RAM, Swap, Free memory,... etc.
  * Average CPU load for past 1min, 5mins and 15 mins.
  * Network Download/Upload rates of network connections.
  * Total number of processes, active ones, sleeping processes etc.
  * Disk I/O related (read or write) speed details
  * Currently mounted devices disk usages.
  * Top processes CPU/Memory usages, Names and file path of exec
* official packages for Debian/RedHat/...

### USSAGE:
  ```
  | $ glances
  |   GREEN : OK       (everything is fine)
  |   BLUE  : CAREFUL  (need attention)
  |   VIOLET: WARNING  (alert)
  |   RED   : CRITICAL (critical)
  |
  | Shortcuts:
  | a – Sort processes automatically
  | c – Sort processes by CPU%
  | m – Sort processes by MEM%
  | p – Sort processes by name
  | i – Sort processes by I/O rate
  |
  | d – Show/hide disk I/O stats ols
  | f – Show/hide file system statshddtemp
  | n – Show/hide network stats
  | s – Show/hide sensors stats
  | y – Show/hide hddtemp stats
  | l – Show/hide logs
  | b – Bytes or bits for network I/Oools
  | w – Delete warning logs
  | x – Delete warning and critical logs
  | x – Delete warning and critical logs
  | 1 – Global CPU or per-CPU stats
  | h – Show/hide this help screen
  | t – View network I/O as combination
  | u – View cumulative network I/O
  | q – Quit (Esc and Ctrl-C also work)
  |
  | default thresholds: /etc/glances/glances.conf*
  | (careful=50, warning=70 and critical=90)
  ```

* Client/Server mode:
  ```
  | ON THE SERVER                       │ ON THE CLIENT
  | ────────────────────────────────────┼────────────────────────────────
  |                                     │
  | # glances -s -B $ADDRESS -p $PORT   │ # glances -c -P 172.16.27.56 *
  |                 (0.0.0.0)   (61209) │
  | Define password for server          │
  | ...
  | Glances server is running on ...
  ```
[[}]]

[[{MONITORING.I/O,MONITORING.STORAGE,PM.TODO]]
## iostat(CPU, I/O, FS)

* <https://linux.die.net/man/1/iostat>
  ```
  | $ iostat -xt 2  # -x Show extended statistics
  |                 # -t Print time
  | avg-cpu:  %user   %nice %system %iowait  %steal   %idle
  |            4.27    0.00    4.27    2.26    0.00   89.20
  |
  | Device  r/s  w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await  w_await aqu-sz rareq-sz wareq-sz svctm %util
  | sda    0.00 5.50  0.00 36.00   0.00   5.00  0.00 47.62    0.00    17.55   0.09     0.00     6.55  0.82  0.45
  | sdb    0.00 0.00  0.00  0.00   0.00   0.00  0.00  0.00    0.00     0.00   0.00     0.00     0.00  0.00  0.00
  | dm-0   0.00 3.00  0.00 12.00   0.00   0.00  0.00  0.00    0.00   123.00   0.37     0.00     4.00  0.50  0.15
  | dm-1   0.00 0.00  0.00  0.00   0.00   0.00  0.00  0.00    0.00     0.00   0.00     0.00     0.00  0.00  0.00
  | dm-2   0.00 7.50  0.00 30.00   0.00   0.00  0.00  0.00    0.00     0.47   0.00     0.00     4.00  0.40  0.30
  |        ^^^^ ^^^^  ^^^^ ^^^^^   ^^^^   ^^^^  ^^^^  ^^^^    ^^^^     ^^^^   ^^^^     ^^^^     ^^^^  ^^^^  ^^^^
  |        read                                            r_await  r_await  aqu-sz                  ignore elapsed
  |        req.                                            avg msec avg msec avg                     to be  time %
  |                                                        for read for read queue                   remove during
  |                                                        requests requests length                         which
  |                                                                          of req                         I/O req
  |                                                                          issued                         were
  |                                                                                                         issued
  |                                                                                                         BANDWIDTH
  |                                                                                                         USSAGE
  ```
[[}]]


## powertop  [[{monitoring.hardware]]

Allows to:
* diagnose device/CPU power consumption issues.
* Tune/control device/CPU power management.

  ```
  | $ sudo powertop # ← Interactive mode if no other option is provided
  | $ sudo powertop --auto-tune  # ← Callibrate non-interactively
  |        ^^^^^^^^^^^^^^^^^^^^
  |        To enable at system boot add next systemd Unit:
  |        *STEP 1: Create/Edit powertop.service like:*
  |        $ sudoedit /etc/systemd/system/powertop.service
  |        (Add next lines)
  |      + [Unit]
  |      + Description=Powertop auto-tune
  |      +
  |      + [Service]
  |      + ExecStart=/usr/bin/powertop --auto-tune
  |      + RemainAfterExit=true
  |      +
  |      + [Install]
  |      + WantedBy=multi-user.target
  | 
  |        *STEP 2: Enable the new service like:*
  |        $ sudo systemctl daemon-reload
  |        $ sudo systemctl enable powertop
  |        $ sudo systemctl start powertop
  | 
  |        *STEP 3: Check it has run properly*
  |        $ sudo journalctl -u powertop
  |      → ...
  |      → systemd[1]: Started Powertop auto-tune.
  |      → powertop[4778]: modprobe cpufreq_stats failedLoaded 0 prior measurements
  |      → powertop[4778]: RAPL device for cpu 0
  |      → powertop[4778]: RAPL Using PowerCap Sysfs : Domain Mask d
  |      → powertop[4778]: RAPL device for cpu 0
  |      → powertop[4778]: RAPL Using PowerCap Sysfs : Domain Mask d
  |      → powertop[4778]: Devfreq not enabled
  |      → powertop[4778]: glob returned GLOB_ABORTED
  |      → powertop[4778]: *Leaving PowerTOP*
  | 
  | OTHER PERTINENT OPTIONS:
  | --calibrate    :  Runs  in  calibration  mode: When running on battery,
  |                  powertop can track power consumption as well as system
  |                  activity.
  |                   When there are enough measurements, powertop can start
  |                  to report  power  estimates.
  | -csv=file      : Generate CSV report.
  | -html=file     : Generate an HTML report.
  | -extech=$USBDEV: Use Extech Power Analyzer for analysis
  |                  USBDEV will be a USB adaptor similar to /dev/ttyUSB0
  | -iteration=$Num : Number of times to run each test.
  | -time[=seconds] : Generate report for specified number of seconds.
  | -workload=file  : Execute workload file as a part of calibration
  | ....
  ```
[[}]]

[[linux.101.process_model}]]

# Network  [[{network]]

[[{configuration.network,troubleshooting.network]]
## NetworkManager
* <https://www.redhat.com/sysadmin/becoming-friends-networkmanager>
* <https://linux.die.net/man/8/networkmanager>
* <https://linux.die.net/man/5/networkmanager.conf>
* <https://linux.die.net/man/1/nmcli>
* <https://linux.die.net/man/8/networkmanager_selinux>

* widespread network configuration daemon
* Managed through cli (nmcli), text-GUI (nmtui) GUI (GNOME,...)
  files, web-console (Cockpit) or D-Bus interface.
  APIs and a library (libnm) is also provided.
-*NetworkManager allows users and applications to retrieve *
 *and modify the network's configuration at the same time, *
 *ensuring a consistent and up-to-date view of the network.*

* NetworkManager philosophy:
  "...attempts to make networking configuration and operation as
    painless and automatic as possible..."
  When there is partial or no configuration, NetworkManager checks
  the available devices and tries its best to provide connectivity
  to the host.

* NetworkManager allows advanced network administrators to
  provide their own configuration.

- *NetworkManager Entities*:

  ```
  | - device     : represents a network interface ("ip link")
  |                A NetworkManager devices tracks:
  |              - If it is managed by NetworkManager
  |              - The available connections for the device
  |              - The connection active on the device (if any)
  | - connection : represents the full configuration to
  |                be applied on a device and is just a list of
  |                properties.
  |                Properties belonging to the same configuration
  |                area are grouped into settings:
  |                Example:
  |              - ipv4 setting group:
  |                - addresses
  |                - gateway
  |                - routes
  | 
  |  ^^^^^^^^^^^^^
  |  *NETWORK SETUP == activate a connection with a device*
  ```

  ```
  |  $ nmcli device   # ← list the devices detected by NetworkManager
  |  (output will be similar to)
  |> DEVICE   TYPE      STATE           CONNECTION
  |> enp1s0   ethernet  connected       ether-enp1s0
  |> enp7s0   ethernet  disconnected    --
  |> lo       loopback  unmanaged  --
  ```

 ```
 | $ nmcli device \        # ← turn off management
 |   set enp1s0 managed no     for enp1s0 device
 |                             (change is not persisted
 |                              and ignored on reboot)
 ```

 ```
 | $ nmcli   # List detailed connections for devices
 | enp1s0: connected to enp1s0
 |       "Red Hat Virtio"
 |       ethernet (virtio_net), 52:54:00:XX:XX:XX, hw, mtu 1500
 |       ip4 default
 |       inet4 192.168.122.225/24
 |       route4 0.0.0.0/0
 |       route4 192.168.122.0/24
 |       inet6 fe80::4923:6a4f:da44:6a1c/64
 |       route6 fe80::/64
 |       route6 ff00::/8
 | ...
 ```

 ```
$ nmcli connection # ← list the available connections
→ NAME         UUID          TYPE       DEVICE
→ ether-enp1s0 23e0d89e-...  ethernet   enp1s0
→ ...
 ```

* To deconfigure the associated device, just instruct NetworkManager
to put the connection down. For instance, to deactivate the
ether-enp1s0 connection:

  ```
  | $ nmcli connection \  # ← deactivate connection
  |     down ether-enp1s0     (deconfigure associated
  |                           device)
  | 
  | $ nmcli connection \  # ← Reactivate
  |     up ether-enp1s0
  | 
  | $ nmcli connection \  # ← Show connection details
  |   show ether-enp1s0
  | connection.id:                  *ether-enp1s0*   ← human readable name
  | connection.uuid:                 23e0d89e-...
  | connection.stable-id:            --
  | connection.type:                 802-3-ethernet ← ethernet, wifi, bond!!, vpn, ...
  | connection.interface-name:       enp1s0         ← binds(restrict) to specific device
  | connection.autoconnect:          yes
  | ...
  | ipv4.method                      auto           ← one of:
  |                                                   auto(DHCP)
  |                                                   manual(static IP in ipv4.addresses),
  |                                                   disabled, link local, shared
  | ipv4.addresses                   192.168.1.201/24
  | ipv4....
  | dhcp4.option[1]                  broadcast_address = 192.168.1.255
  | dhcp4....
  | [...]
  | (See `man nm-settigs` for full info about available parameters)
  ```

  ```
  | $ nmcli connection \               ← Permanently change the connection
  |     modify  *ether-enp1s0* \
  |     ipv4.method manual
  |     ipv4.addresses 10.10.10.1/24 \
  |     ipv4.gateway 10.10.10.254 \
  |     ipv4.dns 10.10.10.254
  | $ nmcli connection up ether-enp1s0 ← New settings will only be applied
  |                                      on connection (re)activation
  | 
  | $ nmcli connection \            # ← avoid activation by NetworkManager
  |     modify ether-ens1s0 \           (you would have to activate manually)
  |     connection.autoconnect no
  | 
  | $ nmcli con add \            ← Create new connection
  |   type ethernet \              - DHCP will be used if no IPv4 config
  |   ifname enp0s1 \                is provided.
  |   con-name  *enp0s1_dhcp* \      (ipv4.method defaults to auto)
  |   autoconnect no               - Run interactively with '--ask' option
  | $ nmcli con                  ← Verify new connection
  | →  NAME         UUID         TYPE     DEVICE
  | →  ...
  | → *enp0s1_dhcp* 64b499cb-... ethernet --
  | →  ...
  | 
  | $ nmcli con edit   ← Interactive editor-mode with inline help will open
  ```

### nmcli TODO:

* "Many features deserve separate blog posts"
  * dispatcher scripts
  * connectivity checkers
  * split DNS
  * MAC address randomization
  * hotspot configuration
  * automatic configuration.

* Check Network status:
  ```
  | <https://linux.die.net/man/1/nm-online>
  | ~:$ nm-online -q   ←     --timeout=10       (Defaults to 30secs)
  | ~:$ echo $?              --exit             Exit immediately if nm is not running or connecting
  | → 0                      --quiet            Don't print anything
  |                          --wait-for-startup Wait for nm startup instead of a connection
  ```

### Troubleshooting Network Manager

  ```
  | bypass network manager by manually launching dhcp client:
  | 
  | $ sudo dhclient eth0 # eth0 stands for "Ethernet 0". Name can differ
  |                        execute 'ip link' to show all network devices
  ```


### ERROR: "Network disconnects after a few seconds"

* Try disabling ModemManager service:

  ```
  | $ sudo systemctl stop    ModemManager.service
  | $ sudo systemctl disable ModemManager.service
  ```
[[configuration.network}]]

[[{linux.101,configuration.network,]]

## TCP/IP Utilities

  ```
  | $ host (ip_address|domain_name) # <· Performs DNS lookup of an internet address
  |                                      (using the Domain Name System, DNS).
  |
  | $ dig  www.amazon.com           # <· Query through (configured) DNS server
  | $ dig -x 10.10.10.10            # <· reverse query to DNS
  | $ dig www.amazon.com @IP.DNS.1  # <· Query to a given DNS server (vs default one)
  | (check man page for more options)
  |
  | $ wget http://www.mydomain.com/myPage # <· HTTP client
  | Options:
  | -m: archive/"m"irrow single web-site
  | -nc: (no clobber) avoid overwriting local files
  |
  | $ wget --spider \
  |   --force-html \
  |   -i bookmarks.html             # <· parse bookmarks.html for links
  | (see man page for more info)
  |
  | $ curl  #  Script oriented HTTP client. (See curl dedicated page)
  | $ curl -M # <· Access the full/huge manual
  ```
[[}]]

[[{monitoring.network.nethogs,monitoring.jobs,troubleshooting.network]]
## Nethogs: bandwidth "top" per ps

* Nethogs is a command line utility for linux that displays the network
  bandwidth used by each application or process in realtime.

  ```
  | $ sudo nethogs                                                                   *
  | ...
  |   PID USER     PROGRAM                      DEV        SENT      RECEIVED
  | 2367  enlighten/opt/google/chrome/chrome    eth0       3.341      20.948 KB/sec
  | 2196  enlighten/usr/lib/firefox-7.0.1/fire  eth0       0.871       0.422 KB/sec
  | 3723  enlighten/usr/bin/pidgin              eth0       0.028       0.098 KB/sec
  | 2206  enlighten/usr/bin/skype               eth0       0.033       0.025 KB/sec
  | 2380  enlighten/usr/lib/chromium-browser/c  eth0       0.000       0.000 KB/sec
  | 0     root     unknown TCP                             0.000       0.000 KB/sec

  |   TOTAL                                                4.274      21.493 KB/sec"
  ```

# Remote access

[[{security.remote_access.ssh,security.aaa,security.audit.user]]

## ssh
* ssh stands for "Secure shell". It is also an onomatopoeia.
* <https://en.wikipedia.org/wiki/Secure_Shell>.
* standard way to access Linux&Other OSes remotely.
* A running "sshd" (ssh daemon) must be instaled and running
  on the remote machine. Used also for tunneling TCP connections.

* Quickly connect to remote machine runnin sshd:
  (Remote machine must allow passwords login)

  ```
  | $ ssh myUser@myRemoteMachine
  | myUser@mYRemoteMachine's password:
  | (enter password to log-in to text terminal)
  ```

  ```
  | Example ~/.ssh/config
  | Host remoteHostAlias1 alias2 ...     <· allows to ssh alias2
  |    HostName 10.230.11.10
  |    ProxyCommand /usr/bin/corkscrew 10.10.10.10 8080 %h %p  ¹
  |    Port 12345                        <· 22 by default
  |    User myRemoteUser                 <· defaults to current active local user
  |    LocalForward   5555 removeIP:3333 <· Forward any local TCP conection to port 5555
  |                                        to port 3333 at remoteIP (as seen by sshd server)
  |                                        (ussually remoteIP == localhost)
  |    RemoteForward 13389 localIP:3389  <· Forward any TCP conection to port 5555 in sshd server
  |                                        to port 3389 in localIP (as seen by ssh client)
  |                                        (ussually localIP == localhost)
  |    TCPKeepAlive true                 <· "sort of ping" to avoid closing connection.
  |
  |    ¹ Optional: "bypass firewall through HTTP proxy
  ```

### ssh Passwordless authentication:

* Useful to execute remote tasks automatically.

1. STEP 01: generate local private secret (key) and associated public key.

  ```
  | $ ssh-keygen
  | (WARN: leave passphrase blank to allow for automated task, in such case
  |  it is better to "hide" the local private key to "neighbours".
  ```
2. STEP 02: Copy associated public key to remote machine
  ```
  | $ ssh-copy-id myRemoteUser@myRemoteMachine
  |   └──────────────────┬───────────────────┘
  |   Will append the public key to
  |     /home/myRemoteUser/.ssh/authorized-keys
  ```

* Now it must be possible to ssh into the remote machine with no password
  ```
  | $ ssh myRemoteUser@myRemoteMachine
  ```

* Troubleshooting passwordless access:
  ```
  | local  $ chmod go-rw ~/.ssh/*  # <· Fix file permission. ssh is paranoid about it.
  | remote $ chmod go-rw ~/.ssh/*  # <· Fix file permission. ssh is paranoid about it.
  ```

* CONTROL SSH:

  ```
  | $ ssh ... -S ${CONTROL_SOCKET} ... host1
  |           └────────┬─────────┘
  |           Later on we can just
  |           ┌────────┴─────────┐
  | $ ssh     -S ${CONTROL_SOCKET} -O $ctr_cmd
  |                                   ^^^^^^^^
  |                       check   : check master process is running.
  |                       forward : request forwardings without command execution.
  |                       cancel  : cancel forwardings
  |                       exit    : request master to exit
  |                       stop    : request the master to stop accepting
  |                                  further multiplexing requests.
  | ...
  | if [[ $6 == "stop" ]] ; then
  |    TUNNEL="${TUNNEL} -O exit "
  | fi
  | TUNNEL="${TUNNEL} $4@$2"
  | $TUNNEL 1>/dev/null
  ```

* ssh client and daemon have a wide range of useful options, including
  reconnection, tunneling, HTTP-sock-proxy-establishment, ...
  See 'man 1 ssh' and 'man 8 sshd' for details.

* Related:
  <https://www.reddit.com/r/linuxadmin/comments/b9c7lw/using_a_yubikey_as_smartcard_for_ssh_public_key/>
[[security.remote_access.ssh}]]

[[{security.remote_access.tmux]]

## Tmux

* Tmux helps to keep your session open when the ssh network connection
  goes down.
* It also allows to split the terminal in panes and "project trees".

[[{security.audit.user,use_case.tutorials]]
* recording/auditing tmux sessions with ASCIInema:
  <https://github.com/asciinema/asciinema/wiki/Recording-tmux-session>
[[security.audit.user}]]

* Oh my tmux!. pretty and versatile tmux configuration
  <https://github.com/gpakosz/.tmux>

### 10 killer tmux tips

* <https://www.sitepoint.com/10-killer-tmux-tips/>

* tmux Plugin Manager:
  ```
  | - PRE-SETUP install:
  |   $ git clone https://github.com/tmux-plugins/tpm \
  |     ~/.tmux/plugins/tpm
  | 
  |   $ cat << EOF >> ~/.tmux.conf
  |   # Init plugin manager.
  |   run -b '~/.tmux/plugins/tpm/tpm'    ←  keep line at the
  |   EOF                                    "very bottom"

  | - Installing Plugins:
  |   STEP 1) ~/.tmux.conf:
  |   ...
  |   set -g @plugin 'tmux-plugins/tmux-resurrect'
  |   set -g @plugin 'tmux-plugins/tmux-continuum'
  |   #               └─────┬────┘ └─────┬──────┘
  |   #               Github user   Github pass.

  |   STEP 2) "prefix" + I (capital i) to fetch the plugin.
  |   (plugins will be cloned to ~/.tmux/plugins/ and then sourced).

  | - Uninstalling Plugins:
  |   STEP 1) Remove / comment out plugin from ~/.tmux.conf
  |   STEP 2) Press "prefix" + alt + u (lowercase u to uninstall)
  ```

Interesting plugins include:

- "Resurrecting tmux": (across system restarts!!!).
  - To restore also vim sessions:
    - PRE-SETUP) install Tim Pope's vim-obsession (vim pluging).
      `cd ~/.vim/bundle`
    - tell vim-obsession to track the session.
  - Ussage:
    - save    tmux session: "prefix" + Control + s
    - restore tmux session: "prefix" + Control + r (run tmux first)
- "tmux-continuum": continuously saves tmux sessions at regular
  intervals, Automatically restores at restart.
- "Zooming tmux": (tmux 1.8+) very useful to look at test failures or
  inspect logs.
  - Ussage:
  - swithc on/off zoom into a pane: "prefix" + z
- ...

[[security.remote_access.tmux}]]


## VNC Remote Desktop [[{security.remote_access.vnc,]]

* The VNC protocol allows to launch a graphical system in a
  remote system using the VNC server and access it remotely
  using the VNC clients.

* Running programs in the remote machine will "draw" to a
  local memory buffer shared by the VNC server.
  When a remote VNC client connects, the VNC server will
  transmit to the client the graphic buffer and finally
  the VNC client will show the buffer in the local display.
* Modern VNC client←→server protocols are highly optimized
  to save bandwith and allows for high-resolution displays
  with around 1Megabit of bandwidth.

* There is no limit to the number of remote VNC servers that
  can run in parallel, just limited by memory access.
  Next links offers an example of how to run many different
  VNC servers (1 or more desktops per user) in a single VNC server:
  <https://github.com/earizon/tigervnc_remote_desktop>
[[security.remote_access.vnc}]]


# SYSTEM INFO 

## Basic System Info

  ```
  [[{linux.101,monitoring.101}]]
  | $ uptime  ← shows how long the computer has been "up" since last reboot.
  |             number of users and the processor load
  | $ date    ← current date/time
  | $ cal     ← display calendar
  | $ uname   ← print information on the system such as OS type, kernel version...
  |             -a --- print all the available information
  |             -m --- print only information related to the machine itself
  |             -n --- print only the machine hostname
  |             -r --- print the release number of the current kernel
  |             -s --- print the operating system name
  |             -p --- print the processor type
  | 
  | $ cat /etc/release | sort | uniq  ← Shows OS identification
  |                                   (Distribution, major,minor,patch,flavour, ...)
  | $ free    ← human-readable memory report (-g: Gigabytes)
  |               total        used        free      shared  buff/cache   available
  | Mem:        6102476      812244     4090752       13112     1199480     4984140
  | Swap:       2097148           0     2097148
  | 
  | $ getconf -ag  ← Get all system config. parameters
  | → ...
  | → PAGESIZE                           4096
  | → ..
  | → ULONG_MAX                          18446744073709551615
  | → USHRT_MAX                          65535
  | → ...
  | → _POSIX_...
  | → ...
  | → LFS_CFLAGS
  | → LFS_LDFLAGS
  | → LFS_LIBS
  | → LFS_LINTFLAGS
  | → LFS64_CFLAGS                       -D_LARGEFILE64_SOURCE
  | → LFS64_LDFLAGS
  | → LFS64_LIBS
  | → LFS64_LINTFLAGS                    -D_LARGEFILE64_SOURCE
  | → ...
  | → GNU_LIBC_VERSION                   glibc 2.28
  | → GNU_LIBPTHREAD_VERSION             NPTL 2.28
  | → POSIX2_SYMLINKS                    1
  | → LEVEL1_ICACHE_SIZE                 32768
  | → LEVEL1_ICACHE_ASSOC                8
  | → LEVEL2_CACHE_SIZE                  262144
  | → LEVEL2_CACHE_ASSOC                 8
  | → LEVEL2_CACHE_LINESIZE              64
  | → LEVEL3_CACHE_SIZE                  3145728
  | → LEVEL3_CACHE_ASSOC                 12
  | → LEVEL3_CACHE_LINESIZE              64
  | → LEVEL4_CACHE_SIZE                  0
  | → LEVEL4_CACHE_ASSOC                 0
  | → LEVEL4_CACHE_LINESIZE              0
  | → ...
  | 
  | $ dmidecode types    <··· Decode DMI (system board)
  |  0 BIOS
  |  1 System
  |  2 Baseboard
  |  4 Processor
  |  6 Memory Module
  |  7 Cache
  | 11 OEM Strings
  | 12 System Config.Options
  | 16 Physical Memory Array
  | 17 Memory Device
  | 24 Hardware Security
  | 25 System Power Controls
  | 32 System Boot
  | 33 64-bit Memory Error
  | 39 Power Supply
  |    ...
  | 
  | $ sudo dmidecode -t 17  <··· Show physical mem.banks
  | | OUPUT PHYSICAL MACHINE                      | OUPUT VIRTUAL MACHINE (Manufacturer: QUEMU,...)
  | | # dmidecode 3.2                             | # dmidecode 3.0
  | | Getting SMBIOS data from sysfs.             | Getting SMBIOS data from sysfs.
  | | SMBIOS 2.7 present.                         | SMBIOS 2.8 present.
  | |                                             |
  | | Handle 0x0008, DMI type 17, 34 bytes        | Handle 0x1100, DMI type 17, 40 bytes
  | | Memory Device                               | Memory Device
  | |     Array Handle: 0x0007                    |         Array Handle: 0x1000
  | |     Error Information Handle: Not Provided  |         Error Information Handle: Not Provided
  | |     Total Width: 64 bits                    |         Total Width: Unknown
  | |     Data Width: 64 bits                     |         Data Width: Unknown
  | |     Size: 8192 MB                           |         Size: 4096 MB
  | |     Form Factor: SODIMM                     |         Form Factor: DIMM
  | |     Set: None                               |         Set: None
  | |     Locator: ChannelA-DIMM0                 |         Locator: DIMM 0
  | |     Bank Locator: BANK 0                    |         Bank Locator: Not Specified
  | |     Type: DDR3                              |         Type: RAM
  | |     Type Detail: Synchronous                |         Type Detail: Other
  | |     Speed: 1333 MT/s                        |         Speed: Unknown
  | |     Manufacturer: Samsung                   |         Manufacturer: QEMU
  | |     Serial Number: 939BED25                 |         Serial Number: Not Specified
  | |     Asset Tag: None                         |         Asset Tag: Not Specified
  | |     Part Number: M471B1G73DB0-YK0           |         Part Number: Not Specified
  | |     Rank: Unknown                           |         Rank: Unknown
  | |     Configured Memory Speed: 1333 MT/s      |         Configured Clock Speed: Unknown
  | |                                             |         Minimum Voltage: Unknown
  | | Handle 0x0009, DMI type 17, 34 bytes        |         Maximum Voltage: Unknown
  | | Memory Device                               |         Configured Voltage: Unknown
  | |     Array Handle: 0x0007                    |
  | |     Error Information Handle: Not Provided  |
  | |     Total Width: 64 bits                    |
  | |     Data Width: 64 bits                     |
  | |    *Size: 8192 MB            *              |
  | |    *Form Factor: SODIMM      *              |
  | |    *Set: None                *              |
  | |    *Locator: ChannelB-DIMM0  *              |
  | |    *Bank Locator: BANK 2     *              |
  | |    *Type: DDR3               *              |
  | |    *Type Detail: Synchronous *              |
  | |    *Speed: 1333 MT/s         *              |
  | |    *Manufacturer: 04CB       *              |
  | |     Serial Number: A8750300                 |
  | |     Asset Tag: None                         |
  | |     Part Number:                            |
  | |     Rank: Unknown                           |
  | |     Configured Memory Speed: 1333 MT/s      |
  ```



[[{security.backup,linux.101,storage.compression]]
# BACKUPS  

[[{security.backups.bup,security.101,]]
## bup: Fast incremental backups with deduplication 

* <https://github.com/bup/backup>
  Very efficient backup system based on git packfile format,
  providing fast incremental saves and global deduplication (among and
  within files, including virtual machine images).

* Data is "automagically" shared between incremental backups without 
  having to know which backup is based on which other one - even if the 
  backups are made from two different computers that don't even know 
  about each other. You just tell bup to back stuff up, and it saves 
  only the minimum amount of data needed
[[security.backups.bup}]]

## CLP  [[{monitoring.logs.clp,security.audit,scalability.storage,PM.low_cost]]

* <https://github.com/y-scope/clp>

* C&P from <https://www.uber.com/en-DE/blog/reducing-logging-cost-by-two-orders-of-magnitude-using-clp/>
  """CLP reduces log size by x169 ...
     CLP allows to search directly on the compressed logs .... With
     just Phase 1, this cost is reduced to $10,000/year for a one month
     retention period. But more importantly, it now enables us to retain
     the logs as our engineers requested: we were able to restore the
     logging level from WARN back to INFO, increase our retention period
     by 10x, and still reduce our storage costs by 17x."""
[[monitoring.logs.clp}]]

## tar (tape-archive) 
* man 1 tar
* standard tool for archiving saving a (many) files and directories to a single
  tape or disk archive.
* Individual files can be restored from the .tar file when needed.

Examples:

* Create new archive file from directory
  ```
  | $ INPUT="/home/myUser/myProject"            # directory/ies and/or file/s to archive.
  | $ OUTPUT="_home_myUser_myProject_01.tar.gz" # prefixing with absolute path is optional.
  |
  | $ tar czf ${OUTPUT} ${INPUT}
  |       ·└─ pipe tar output as input for g(z)ip compressor.
  |       ·   alt: (j) for bzip2 (better and slower compression)
  |       └─· (c)oncatenate (append to tar)  files ${INPUT} 
  ```

* Restore backup from archive file:
  ```
  | $ cd /home/myUser
  | $ tar xzf _home_myUser_myProject_01.tar.gz
  |       ││└·· name of input file
  |       │└─·· de-compress with gzip first
  |       └──·· e(x)tract 
  ```

* List contents of archive:
  ```
  | $ tar tzf _home_myUser_myProject_01.tar.gz
  |       ││└·· name of  input file
  |       │└─·· de-compress with gzip first
  |       └──·· list
  ```

[[{security.backup.incremental,storage.network]]
## Remote Incremental Backup

* EasyUp: KISS incremental remote backup around rsync+ssh
  <https://github.com/earizon/easyup>

* Rsnapshot: filesystem snapshot utility on top of rsync.
  <http://rsnapshot.org/><br/>
  """rsnapshot makes it easy to make periodic snapshots of local 
  machines, and remote machines over ssh.  The code makes extensive use 
  of hard links whenever possible, to greatly reduce the disk space 
  required and rsync to save bandwidth (backup only changes)"""

* Live backups with inotify + rsync + bash: Backup on "real-time changes"<br/>
  <https://linuxhint.com/inotofy-rsync-bash-live-backups/>

* <http://www.bacula.org/>
  """Bacula is a set of Open Source, computer programs that permit to manage backup,
     recovery, and verification of computer data across a network of computers of different
     kinds,  offering many advanced storage management features that make it
     easy to find and recover lost or damaged files."""
     -<http://www.bacula.org/9.0.x-manuals/en/main/index.html>
  * *Director Daemon* supervises all the backup, restore, verify and archive operations.
    Sysadmin uses Director to schedule backups and to recover files..
  * *Console service* allows the administrator or user to communicate with the Director
    (three versions: text-based, QT-based, wxWidgets)
  * *File Daemon* It's installed on the machine to be backed up and is responsible for
    providing the file attributes and data when requested by the Director
    as well as for the file system dependent part of restoring the file attributes and data
    during a recovery operation.
  * *Storage daemons* are software programs in charge of storage and recovery of the
    file attributes and data to the physical backup media or volumes. In other words, it is
    responsible for reading and writing your tapes (or other storage media, e.g. files)
  * *Catalog Services* are responsible for maintaining the file indexes and
    volume databases for all files backed up allowing sysadmin or user to
    quickly locate and restore any desired file. The Catalog services sets
    Bacula apart from simple backup programs like tar and bru, because the catalog
    maintains a record of all Volumes used, all Jobs run, and all Files saved, permitting
    efficient restoration and Volume management. Bacula currently supports three different
    databases, MySQL, and PostgreSQL one of which must be chosen when building Bacula.
  * *Monitor Service* Allows the administrator or user to watch current status of Directors,
    File Daemons and Bacula Storage Daemons. Currently, only a GTK+ version is available.

* Symple remote backups with ssh
  ```
  $ tar cjf - myDirToBackup \       # local
    | ssh myUser@myRemoteMachine \  # ssh pipe
    "cd myBackupPath && tar -xjf -" # remote
  ```

* "Real time" backup with rsync and bash
  <https://github.com/Leo-G/backup-bash>

[[}]]

## rsync

* Standard utility to copy files remotely by just comparing local/remote changes
  (vs copying the data like a full).
* It its at the core of many other backups tools.

  ```
  Example. 
  $ cd ...
  $ rsync --dry-run \                         <· Simulate execution, remove to really execute
    --links \
    --stats \
    --itemize-changes \
    --recursive \                             <· mandatory for directories
    --human-readable \
    --log-file=rsync_dry_run.log \
    some/Local/FolderOrFile \
     user@host:/some/Remote/FolderOrFile      <· use ssh to connect and launch remote rsync peer
  ```

[[{storage.compression,scalability.backups,performance.CPU,doc_has.comparative]]
## multi-core compression tools 

* NOTE: bzip2 offers a much better compression than gz when source
  files have low entropy (many repeated words, white spaces,...) while
  consuming (much) more CPU.<br/>
  Ideal to save bandwidth while highly increasing CPU ussage.

* Ex:
  * two runs to avoid FS cache effects
  * Input: (dir1), 430M, with 70% binary (hard to compress) data.
  ```
  | STANDARD UTILITY                           MULTI-CORE ALTERNATIVE
  | NO COMPRESSION                          |
  | ==================================      |
  | $ sudo tar cf output.tar dir1           |
  |                                         |
  | real  0m0,096s                          |
  | user  0m0,048s                          |
  | sys   0m0,049s                          |
  | OUTPUT SIZE: 410M                       |

  | GZIP                                    |   PIGZ (DEF.OPTIONS, 4 CORES)
  | ==================================      |   ==============================
  | $ sudo tar cf - dir1 | \                |   $ sudo tar cf - directory1 | \
  |    gzip  > output.tar.gz                |         pigz  > /dev/null
  | TIME: 13,5s                             |   TIME:  5,0s
  | OUTPUT SIZE: 337M                       |   OUTPUT SIZE: 337M

  | BZIP2                                   |   PBZIP2 (DEF.OPTIONS, 4 CORES)
  | ==============================          |   =================================
  | $ sudo tar cf dir1 | \                  |   $ sudo tar cf - dir1 | \
  |    bzip2 > output.tar.bz2               |    pbzip2 > output.tar.bz2
  | TIME: 66,0s                             |   TIME: 31,8s
  | OUTPUT SIZE: 331M                       |   OUTPUT SIZE: 332M


  | XZ                                      |   PIXZ  (DEFAULT OPTIONS, 4 CORES)
  | ==============================          |   =================================
  | $ sudo tar cf dir1 | \                  |   $ sudo tar cf - dir1 | \
  |    xz    > output.tar.bz2               |     pbzip2 > output.tar.bz2
  | TIME: 151,7s                            |   TIME: 46,8s
  | sys     0m2,179s                        |   sys     0m1,706s
  | OUTPUT SIZE: 313M                       |   OUTPUT SIZE: 317M
  ```

From: <https://www.linuxlinks.com/best-linux-multi-core-compression-tools/>
* pigz    Parallel implementation of gzip. It's a fully functional replacement for gzip
* PBZIP2  Parallel implementation of the bzip2 block-sorting file compressor
* PXZ     Runs LZMA compression on multiple cores and processors
* lbzip2  Parallel bzip2 compression utility, suited for serial and parallel processing
* plzip   Massively parallel (multi-threaded) lossless data compressor based on lzlib
* lrzip   Compression utility that excels at compressing large files
* pixz    Parallel indexing XZ compression, fully compatible with XZ. LZMA and LZMA2
[[}]]


## Zstandard, superfast compression  [[{scalability.backups.zstandard,PM.TODO,]]

* <https://www.2daygeek.com/zstandard-a-super-faster-data-compression-tool-for-linux/>
[[]]
[[scalability.backups.zstandard}]]

[[{security.android.101,storage.android]]
## Android To Linux Backups  
* <https://github.com/FlamingTuri/android-file-backup>

Script solving common problems when transferring files/folders from/to Android.
* transfer big folders
* keep the files original modification date

PREREQUISITES
* On Linux:
  * `adb` command installed. It can be downloaded from
    <https://developer.android.com/studio/releases/platform-tools>
    (~10MB aprox for the whole toolkit)
  * <https://developer.android.com/studio/command-line/adb>

  ```
  $ sudo apt-get install pv  # <· Optional. Allows to view compression progress (-z option)
  ```

* On Android:
  * enable USB debugging
  * enable USB file transfer.
    (accept dialog asking to trust your computer RSA fingerprint)

Execution Example:

  ```
  | $ ./adb-backup.sh    ← backup "DCIM", "Download", "Pictures"
  |    /sdcard/DCIM        from Android to Linux FS.
  |    /sdcard/Download
  |    /sdcard/Pictures
  ```
[[security.android.101}]]

[[security.backup}]]

[[{]]
## Linux Boot Process
```
 Power On
 > 1) BIOS/UEFI
   - load from non-volatile memory.
   - run POST
   > 2) Detect Devices
     > 3) Choose a Boot Device
       > 4) GRUB
            Boot Loader
         > 5) Execute SystemD: first process in user space
           > 6) Run .target Files
                · default.target
                · multi-user.target
                  · Basic.target
                  · getty.target
                  · ssh.service
             > 7) Run Startup Scripts
                  ··> /systemd-logind
                   ··> /etc/profile
                    ··> ${HOME}/.bashrc
               > Users can now login
```
[[}]]


[[{linux.101,use_case.*,PM.TODO]]
## GNU Coreutils 9.0

* <https://www.gnu.org/software/coreutils/manual/html_node/index.html>
[[linux.101}]]