add doc

doc/wallet/design/design.md

@@ -0,0 +1,89 @@
+# Grin Wallet + Library Design
+
+![wallet design](wallet-arch.png)
+
+## High Level Wallet Design Overview
+
+The current Grin `wallet` crate provides several layers of libraries, services, and traits that can be mixed, matched and reimplemented to support
+various needs within the default Grin wallet as well as provide a set of useful library functions for 3rd-party implementors. At a very high level,
+the code is organized into the following components (from highest-level to lowest):
+
+* **Command Line Client** - The command line client invoked by `grin wallet [command]`, simply instantiates the other components below
+    and parses command line arguments as needed.
+* **Web Wallet Client** - [Work In Progress]  A web wallet client accessible from the local machine only. Current code can be viewed here:
+  https://github.com/mimblewimble/grin-web-wallet
+* **Static File Server** - [TBD] A means of serving up the web wallet client above to the user (still under consideration)
+* **libWallet** - A high level wallet library that provides functions for the default grin wallet. The functions in here can be somewhat
+  specific to how the grin wallet does things, but could still be reused by 3rd party implementors following the same basic principles as grin
+  does. Major functionality is split into:
+  * **Owner API** - An API that provides information that should only be viewable by the wallet owner
+  * **Foreign API** - An API to communicate with other wallets and external grin nodes
+  * **Service Controller** - A Controller that instantiates the above APIs (either locally or via web services)
+  * **Internal Functions** Helper functions to perform needed wallet tasks, such as selecting coins, updating wallet outputs with
+  results from a Grin node, etc.
+* **libTx** - Library that provides lower-level transaction building, rangeproof and signing functions, highly-reusable by wallet implementors.
+* **Wallet Traits** - A set of generic traits defined within libWallet and the `keychain` crate . A wallet implementation such as Grin's current
+  default only needs to implement these traits in order to provide a wallet:
+  * **NodeClient** - Defines communication between the wallet, a running grin node and/or other wallets
+  * **WalletBackend** - Defines the storage implementation of the wallet
+  * **KeyChain** - Defines key derivation operations
+
+## Module-Specific Notes
+
+A full API-Description for each of these parts is still TBD (and should be generated by rustdoc rather than repeated here). However a few design
+notes on each module are worth mentioning here.
+
+### Web Wallet Client / Static File Server
+
+This component is not a 3rd-party hosted 'Web Wallet' , but a client meant to be run on the local machine only by the wallet owner. It should provide
+a usable browser interface into the wallet, that should be functionally equivalent to using the command line but (hopefully) far easier to use.
+It is currently not being included by a default grin build, although the required listener is currently being run by default. To build and test this
+component, see instructions on the [project page](https://github.com/mimblewimble/grin-web-wallet). The 'Static File Server' is still under
+discussion, and concerns how to provide the web-wallet to the user in a default Grin build.
+
+### Owner API / Foreign API
+
+The high-level wallet API has been split into two, to allow for different requirements on each. For instance, the Foreign API would listen on
+an external-facing port, and therefore potentially has different security requirements from the Owner API, which can simply be bound to localhost
+only.
+
+### libTX
+
+Transactions are built using the concept of a 'Slate', which is a data structure that gets passed around to all participants in a transaction,
+with each appending their Inputs, Outputs or Signatures to it to build a completed wallet transaction. Although the current mode of operation in
+the default client only supports single-user - single recipient, an arbitrary number of participants to a transaction is supported within libTX.
+
+### Wallet Traits
+
+In the current code, a Wallet implementation is just a combination of these three traits. The vast majority of functions within libwallet
+and libTX have a signature similar to the following:
+
+```rust
+pub fn retrieve_outputs<T: ?Sized, C, K>(
+!·wallet: &mut T,
+!·show_spent: bool,
+!·tx_id: Option<u32>,
+) -> Result<Vec<OutputData>, Error>
+where
+!·T: WalletBackend<C, K>,
+!·C: NodeClient,
+!·K: Keychain,
+{  
+```
+
+With `T` in this instance being a class that implements  the `WalletBackend` trait, which is further parameterized with implementations of
+`NodeClient` and `Keychain`.
+
+There is currently only a single implementation of the Keychain trait within the Grin code, in the `keychain` crate exported as `ExtKeyChain`.
+The `Keychain` trait makes several assumptions about the underlying implementation, particularly that it will adhere to a
+[BIP-38 style](https://github.com/bitcoin/bips/blob/master/bip-0038.mediawiki) 'master key -> child key' model.
+
+There are two implementations of `NodeClient` within the code, the main version being the `HTTPNodeClient` found within `wallet/src/client.rs` and
+the seconds a test client that communicates with an in-process instance of a chain. The NodeClient isolates all network calls, so upgrading wallet
+communication from the current simple http interaction to a more secure protocol (or allowing for many options) should be a simple
+matter of dropping in different `NodeClient` implementations.
+
+There are also two implementations of `WalletBackend` within the code at the base of the `wallet` crate. `LMDBBackend` found within
+`wallet/src/lmdb_wallet.rs` is the main implementation, and is now used by all grin wallet commands. The earlier `FileWallet` still exists
+within the code, however it is not invoked, and given there are no real advantages to running it over a DB implementation, development on it
+has been dropped in favour of the LMDB implementation.

doc/wallet/design/goals.md

@@ -0,0 +1,82 @@
+
+Mode of Interactions
+====================
+
+There's a variety of ways wallet software can be integrated with, from hardware
+to automated bots to the more classic desktop wallets. No single implementation
+can hope to accommodate all possible interactions, especially if it wants to
+remain user friendly (who or whatever the user may be). With that in mind, Grin
+needs to provide a healthy base for a more complete wallet ecosystem to
+develop.
+
+We propose to achieve this by implementing, as part of the "standard" wallet:
+
+* A good set of APIs that are flexible enough for most cases.
+* One or two default main mode of interaction.
+
+While not being exhaustive, the different ways we can imagine wallet software
+working with Grin are the following:
+
+1. A receive-only online wallet server. This should have some well-known network
+   address that can be reached by a client. There should be a spending key kept
+   offline.
+1. A fully offline interaction. The sender uses her wallet to dump a file that's
+   sent to the receiver in any practical way. The receiver builds upon that file,
+   sending it back to the sender. The sender finalizes the transaction and sends it
+   to a Grin node.
+1. Fully online interaction through a non-trusted 3rd party. In this mode
+   receiver and sender both connect to a web server that facilitates the
+   interaction. Exchanges can be all be encrypted.
+1. Hardware wallet. Similar to offline but the hardware wallet interacts with
+   a computer to produce required public keys and signatures.
+1. Web wallet. A 3rd party runs the required software behind the scenes and
+   handles some of the key generation. This could be done in a custodial,
+   non-custodial and multisig fashion.
+1. Fully programmatic. Similar to the online server, but both for receiving and
+   sending, most likely by an automated bot of some sorts.
+
+As part of the Grin project, we will only consider the first 2 modes of
+interaction. We hope that other projects and businesses will tackle other modes
+and perhaps even create new ones we haven't considered.
+
+Design Considerations
+=====================
+
+Lower-level APIs
+----------------
+
+Rust can easily be [reused by other languages](https://doc.rust-lang.org/1.2.0/book/rust-inside-other-languages.html)
+like Ruby, Python or node.js, using standard FFI libraries. By providing APIs
+to build and manipulate commitments, related bulletproofs and aggregate
+signatures we can kill many birds with one stone:
+
+* Make the job of wallet implementers easier. The underlying cryptographic
+  concepts can be quite complex.
+* Make wallet implementations more secure. As we provide a higher level API,
+  there is less risk in misusing lower-level constructs.
+* Provide some standardization in the way aggregations are done. There are
+  sometimes multiple ways to build a commitment or aggregate signatures or proofs
+  in a multiparty output.
+* Provide more eyeballs and more security to the standard library. We need to
+  have the wallet APIs thoroughly reviewed regardless.
+
+Receive-only Online Wallet
+--------------------------
+
+To be receive only we need an aggregation between a "hot" receiving key and an
+offline spending key. To receive, only the receiving key should be required, to
+spend both keys are needed.
+
+This can work by forming a multi-party output (multisig) where only the public
+part of the spending key is known to the receiving server. Practically a master
+public key that can be derived similarly to Hierarchical Deterministic wallets
+would provide the best security and privacy.
+
+TODO figure out what's needed for the bulletproof. Maybe pre-compute multiple
+of them for ranges of receiving amounts (i.e. 1-10 grins, 10-100 grins, etc).
+
+Offline Wallet
+--------------
+
+This is likely the simplest to implement, with each interaction dumping its
+intermediate values to a file and building off each other.

doc/wallet/design/wallet-arch.puml

@@ -0,0 +1,110 @@
+@startuml grin-wallet-overview 
+skinparam componentStyle uml2
+
+[Grin Node] as grin_node
+
+folder "Provided by Grin" as services {
+	component foreign_api [
+		**Foreign API**
+		External-Facing functions
+		- receive_tx, build coinbase
+	]
+
+	component owner_api [
+		**Owner API**
+		Functions used by wallet owner only
+		- retrieve outputs, retrieve txs,
+		  get balances, send, etc. . .
+
+	]
+	component libtx [
+        **Transaction Library (libTx)**
+        Lower-Level transaction functions
+        - Build transaction (via Slate), sign,
+         build reward, fees, etc. . .
+	]
+	component libwallet [
+        **Wallet Library (libWallet)	**
+        - Higher level wallet functions (select coins,
+          update wallet from node, etc)
+        - Service Controller
+            (instantiate libs, start listeners)
+	]
+	() "Owner HTTP Listener (localhost only)" as owner_http
+	() "Foreign HTTP Listener" as foreign_http
+	() "Owner Single-Use" as owner_single
+	() "Foreign Single-Use" as foreign_single
+}
+
+' Trait definitions
+package "Traits Implemented by Wallets" as traits {
+	database "WalletBackend" as wallet_backend
+	database "KeyChain" as keychain
+	component "NodeClient" as wallet_client
+}
+
+note left of wallet_client
+    - Communication layer implementation
+    - Handles underlying communication with grin node
+       or other wallets
+    - HTTP implementation provided currently, (Other,
+       more secure protocols possible.)
+end note
+
+note bottom of keychain
+    - Handles all key derivation operations
+end note
+
+note bottom of wallet_backend
+    - Implements underlying storage for wallet data
+    - LMDB  storage provided in default client, others
+      possible (Flat-file, other DBs, etc)
+end note
+
+libtx <--> traits
+libwallet <--> traits
+
+note right of traits
+    **Default Wallet simply a struct that provides**
+    **implementations for these 3 traits**
+end note
+
+' Client Side
+'package "Provided as reference implementation" {
+	[Pure JS Wallet Client Implementation] as js_client
+	[Command Line Wallet Client] as cl_client
+	component web_server [
+			V. Light Rust Web Server - Serve static files (TBD)
+			(Provided by default - localhost only)
+			(Serve up pure JS client)
+	]
+'}
+
+[External Wallets] as external_wallets
+[External Wallets] as external_wallets_2
+
+wallet_client <--> grin_node
+wallet_client <--> external_wallets_2
+
+web_server <--> owner_http
+js_client <-- web_server
+cl_client <--> owner_single
+cl_client <--> foreign_single
+
+owner_single <--> owner_api
+foreign_single <--> foreign_api
+
+libwallet <--> libtx
+
+foreign_api --> libwallet
+owner_api --> libwallet
+
+js_client <--> owner_http
+owner_http <--> owner_api
+external_wallets <--> foreign_http
+foreign_http <--> foreign_api
+
+'layout fix
+'grin_node -[hidden]- wallet_backend
+
+@enduml

doc/wallet/tls-setup.md

@@ -0,0 +1,74 @@
+# Wallet TLS setup
+
+## What you need
+* A server with a static IP address (eg `3.3.3.3`)
+* A domain name ownership (`example.com`)
+* DNS configuration for this IP (`grin1.example.com` -> `3.3.3.3`)
+
+If you don't have a static IP you may want to consider using services like DynDNS which support dynamic IP resolving, this case is not covered by this guide, but all the next steps are equally applicable.
+
+If you don't have a domain name there is a possibility to get a TLS certificate for your IP, but you have to pay for that (so perhaps it's cheaper to buy a domain name) and it's rarely supported by certificate providers.
+
+## I have a TLS certificate already
+Uncomment and update the following lines in wallet config (by default `~/.grin/grin-wallet.toml`):
+
+```toml
+tls_certificate_file = "/path/to/my/cerificate/fullchain.pem"
+tls_certificate_key =  "/path/to/my/cerificate/privkey.pem"
+```
+
+If you have Stratum server enabled (you run a miner) make sure that wallet listener URL starts with `https` in node config (by default `~/.grin/grin-server.toml`):
+
+```toml
+wallet_listener_url = "https://grin1.example.com:13415"
+```
+
+Make sure your user has read access to the files (see below for how to do it). Restart wallet. If you changed your node configuration restart `grin` too. When you (or someone else) send grins to this wallet the destination (`-d` option) must start with `https://`, not with `http://`.
+
+## I don't have a TLS certificate
+You can get it for free from [Let's Encrypt](https://letsencrypt.org/). To simplify the process we need `certbot`.
+
+### Install certbot
+Go to [Certbot home page](https://certbot.eff.org/), choose I'm using `None of the above` and your OS (eg `Ubuntu 18.04` which will be used as an example). You will be redirected to a page with instructions like [steps for Ubuntu](https://certbot.eff.org/lets-encrypt/ubuntubionic-other). Follow instructions from `Install` section. As result you should have `certbot` installed.
+
+### Obtain certificate
+If you have experince with `certboot` feel free to use any type of challenge. This guide covers the simplest case of HTTP challenge. For this you need to have a web server listening on port `80`, which requires running it as root in the simplest case. We will use the server provided by certbot. **Make sure you have port 80 open**
+
+```sh
+sudo certbot certonly --standalone -d grin1.example.com
+```
+
+It will ask you some questions, as result you should see something like:
+
+```
+Congratulations! Your certificate and chain have been saved at:
+  /etc/letsencrypt/live/grin1.example.com/fullchain.pem
+ Your key file has been saved at:
+  /etc/letsencrypt/live/grin1.example.com/privkey.pem
+ Your cert will expire on 2019-01-16. To obtain a new or tweaked
+ version of this certificate in the future, simply run certbot
+ again. To non-interactively renew *all* of your certificates, run
+"certbot renew"
+```
+
+### Change permissions
+Now you have the certificate files but only root user can read it. We run grin as `ubuntu` user. There are different scenarios how to fix it, the simplest one is to create a group which will have access to `/etc/letsencrypt` directory and add our user to this group.
+
+```sh
+sudo groupadd tls-cert
+sudo usermod -a -G tls-cert ubuntu
+chgrp -R tls-cert /etc/letsencrypt
+chmod -R g=rX /etc/letsencrypt
+sudo chmod 2755 /etc/letsencrypt
+```
+
+The last step is needed for renewal, it makes sure that all new files will have the same group ownership.
+
+### Update wallet config
+Refer to `I have a TLS certificate already` because you have it now. Use the folowing values:
+
+```toml
+tls_certificate_file = "/etc/letsencrypt/live/grin1.example.com/fullchain.pem"
+tls_certificate_key =  "/etc/letsencrypt/live/grin1.example.com/privkey.pem"
+```
+