docs: middleware and RPC calls

.gitignore

@@ -32,6 +32,7 @@ tools/bbbconfgen/test/test-output.conf
 /_site
 /docs/.sass-cache
 /docs/_site
+docs/.jekyll-cache/*
 
 # Ignore the binary build outputs (except READMEs)
 /bin/go/*

docs/customapps/middleware/middleware-communication.md

@@ -7,4 +7,47 @@ nav_order: 300
 ---
 ## Middleware: Communication
 
-(TODO)0xB10C
+
+### BitBoxApp <-> Middleware RPCs
+
+TODO
+
+The Middleware handles the communication between the BitBoxApp and the BitBoxBase.
+
+- minimize traffic
+- RPC Client/Server (Go RPC package)
+- Why Websockets
+- End to End encryption
+- Noise paring / handshake
+- TLS vs Noise
+- JWT Authentication
+
+Specific RPC sequence diagrams:
+
+* [Streaming ServiceInfo changes to App frontend](base-streaming-service-info-changes-to-frontend_sequencediagram-org.svg){:target="_blank"}
+
+
+### HSM communication
+
+TODO
+
+### IPC notifications
+
+The Middleware is able to receive IPC notifications from other processes running on the BitBoxBase.
+IPC notifications are implemented via a Unix named pipe and notifications are formatted in a JSON based protocol.
+Using a named pipe allows simple and dependencyless implementation in other Go, Python or Shell Script based processes running on the BitBoxBase.
+One draw back is that a named pipe blocks writes until the content is read.
+By default the named pipe is located in `/tmp/middleware-notification.pipe` and system level permissions are required to write in the pipe.
+
+![middleware ipc notifications](middleware_ipc_notifications.png)
+
+*Schematic showing multiple scripts writing notifications into the named pipe.*
+
+The JSON based protocol for the notifications is versioned, includes a notification topic and a can contain a  payload.
+The payload can contain any valid JSON structure.
+Since a write operation to a unix pipe only being atomic (i.e. two writes do not interleave) as long as the amount written is smaller than the `PIPE_BUF`, which is 4096 bytes for Linux, the Middleware drops all notifications bigger than 4095 bytes.
+
+*Sample notification:*
+```JSON
+{"version": 1, "topic": "sampletopic", "payload": {"sampleInt":123,"sampleString": "string", "sampleBool": true}}
+```

docs/customapps/middleware/middleware-configuration.md

@@ -7,4 +7,48 @@ nav_order: 200
 ---
 ## Middleware: Configuration
 
-(TODO)0xB10C
+
+### Command line interface parameters
+
+Running `bbbmiddleware -h` prints the configuration options for the Middleware on the command line.
+
+Note: *The argument list below was last updated in December 2019.*
+
+```console
+-bbbcmdscript string
+    Path to the bbb-cmd.sh script that allows executing system commands (default "/opt/shift/scripts/bbb-cmd.sh")
+-bbbconfigscript string
+    Path to the bbb-config.sh script that allows setting system configuration (default "/opt/shift/scripts/bbb-config.sh")
+-bbbsystemctlscript string
+    Path to the bbb-systemctl.sh script that allows starting and stopping services on the Base (default "/opt/shift/scripts/bbb-systemctl.sh")
+-datadir string
+    Directory where the Middleware persistent data, like for example the noise encryption keys, is stored (default ".base")
+-electrsport string
+    Electrs RPC port (default "51002")
+-hsmfirmwarefile string
+    Location of the signed HSM firmware binary (default "/opt/shift/hsm/firmware-bitboxbase.signed.bin")
+-hsmserialport string
+    Serial port used to communicate with the HSM (default "/dev/ttyS0")
+-middlewareport string
+    Port the Middleware listens on (default "8845")
+-network string
+    Indicate wether Bitcoin is running on mainnet or testnet (default "testnet")
+-notificationNamedPipePath string
+    Path where the Middleware creates a named pipe to receive notifications from other processes on the BitBoxBase (default "/tmp/middleware-notification.pipe")
+-prometheusurl string
+    URL of the Prometheus server (default "http://localhost:9090")
+-redismock
+    Flag to use the Redis mock for development instead of connecting to a redis server
+-redisport string
+    Port of the Redis server (default "6379")
+-updatehsmfirmware
+    Set to true to force HSM firmware update
+-updateinfourl string
+    URL to query information about Base image updates from (default "https://shiftcrypto.ch/updates/base.json")
+```
+
+### Changing the Middleware systemd file
+
+(TODO)Stadicus
+
+- config in systemd unit (via env variables / config template in /etc/bbbmiddleware)

docs/customapps/middleware/middleware-functionality.md

@@ -7,4 +7,45 @@ nav_order: 100
 ---
 ## Middleware: Functionality
 
-(TODO)0xB10C
+
+The Middleware connects the BitBoxBase system with the BitBoxBase user interfaces, such as the BitBoxApp and the HSM screen.
+The task of the Middleware is to respond to synchronous user request and to asynchronously notify the user about changes on the system.
+A action from the user could for example toggle _SSH access_ or requests for information about the Bitcoin blockchain synchronization progress.
+A notification from the Middleware to the user could be for example about an available software update or the disk running out of space.
+
+### Tasks of the Middleware
+
+The Middleware processes Remote Procedure Calls (RPCs) from the BitBoxApp, keeps a connection to the HSM, queries both Redis and Prometheus servers, executes shell scripts to change system configuration and listens on an inter process communication (IPC) interface for notifications from other processes.
+
+<br>
+![Middleware overview](middleware_overview.png)
+*Schematic showing the Middleware and it's connections.*
+<br>
+<br>
+
+#### App connectivity via RPC
+The BitBoxApp acts as RPC Client talking to the Middleware's RPC Server.
+Available RPCs can for example change system configuration, set the user password, create or restore a backup, update the BitBoxBase software or deliver information on services such as Bitcoin Core or c-lightning running on the Base.
+The [BitBoxApp <-> Middleware communication section](middleware-communication.html) includes more detail on design and implementation decisions.
+
+#### HSM connectivity via UART
+
+The Middleware communicates with the HSM over UART.
+Messages are serialized as Protobuf messages.
+
+#### Querying Redis and Prometheus
+
+The Redis server on the BitBoxBase acts as a store for system configuration and the Prometheus server holds service specific information.
+The Middleware queries information from both.
+
+
+#### Executing shell scripts
+
+The Middleware calls the [bbb-cmd.sh](../bbb-cmd.html) and [bbb-config.sh](../bbb-config.html) scripts to change system configuration and to call standard commands.
+
+
+#### Listening for IPC notifications
+
+Processes running on the BitBoxBase can notify the Middleware about changes the user needs to be made aware of.
+This is for example used to notify the Middleware about a successful completion of system checks after an update.
+The [communication section](middleware-communication.html) includes detailed information about the IPC implementation.

docs/sources/base-pre-update_sequencediagram-org.txt

@@ -0,0 +1,15 @@
+title pre-update
+
+participant app
+
+middleware->shiftcrypto.ch/updates/base.json: HTTP GET
+middleware<-shiftcrypto.ch/updates/base.json: response
+box over middleware: parse JSON
+box over middleware: compare base\nversion with \nbase.json version
+
+
+box over middleware: if newer
+middleware -->> app: notify app about newer version avaliable
+
+middleware <- app: GetUpdateVersion 
+middleware -> app: {\nupdateAvaliable: bool, \nversion: string, \ndescription: string, \nseverity: int\n}

docs/sources/base-streaming-service-info-changes-to-frontend_sequencediagram-org.txt

@@ -0,0 +1,33 @@
+
+title Streaming ServiceInfo changes to the Frontend
+
+participant Frontend
+participant App
+participant Middleware
+participant Prometheus
+participant Prometheus Scraper
+
+
+Middleware->Prometheus: query last scrape timestamp
+Middleware<-Prometheus: timestamp: 2
+
+note over Middleware: compare with saved timestamp\n\nequal:\ndo nothing
+note over Middleware: wait 5 seconds
+
+Prometheus <<-- Prometheus Scraper: write new data\ntimestamp: 3
+Middleware->Prometheus: query last scrape timestamp
+Middleware<-Prometheus: timestamp: 3
+note over Middleware: compare with saved timestamp\n\nnot equal:\n1. safe timestamp = 3\n2. check if ServiceInfo changed
+Middleware->Prometheus: query ServiceInfo data\n(multiple querys)
+Middleware<-Prometheus: data ServiceInfo
+note over Middleware: check if ServiceInfo data changed\n\n- not changed: skip\n- changed: fire event
+App<<--Middleware:notify ServiceInfo changed
+note over Middleware:wait 5 seconds
+Frontend<<--App:notify ServiceInfo changed
+
+Frontend->App: GET /service-info
+Prometheus <<-- Prometheus Scraper: write new data\ntimestamp: 4
+App->Middleware:GetServiceInfo RPC
+App<-Middleware:Return cached\nGetServiceInfoResponse
+Frontend<-App:GetServiceInfoResponse
+note over Frontend:Redraw

docs/sources/base-update-progress-via-rpc_sequencediagram-org.txt

@@ -0,0 +1,39 @@
+title Base Update Process via RPC
+
+note over front-end: user wants to update 
+front-end->App: POST /update-base <version>
+App->Middleware: UpdateBase RPC <version>
+Middleware->bbb-cmd.sh: mender-update install <version>
+
+activate bbb-cmd.sh
+
+bbb-cmd.sh -->> Middleware: read from stdin: progress 1%
+Middleware -->> App: Event: UpdateProgresChanged
+App -->> front-end: Event: UpdateProgressChanged
+note over front-end:Am I ready to query new\nupdate progress info?\n\nNo (do nothing).
+
+bbb-cmd.sh -->> Middleware: read from stdin: progress 2%
+Middleware -->> App: Event: UpdateProgresChanged
+App -->> front-end: Event: UpdateProgressChanged
+note over front-end:Am I ready to query new\nupdate progress info?\n\nYes.
+
+
+front-end->App: GET /base-update-progress
+bbb-cmd.sh -->> Middleware: read from stdin: progress 3%
+
+App->Middleware:GetBaseUpdateProgress
+Middleware->App: 3%
+App->front-end: 3%
+
+
+==when update finished==
+
+bbb-cmd.sh -> Middleware: (no error)
+deactivate bbb-cmd.sh
+
+note over Middleware: set redis keys and restart base 
+
+Middleware -> App: ErrorResponse{Success: true}
+
+App ->front-end: {success: true}
+

docs/sources/middleware_ipc_notifications.drawio

@@ -0,0 +1 @@
+<mxfile host="www.draw.io" modified="2019-12-06T15:58:27.143Z" agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" etag="RTHbtqvwlBRxEknqTNL3" version="12.3.8" type="device" pages="1"><diagram id="dUab36Q9Gzoi0dbqar7h" name="Page-1">3Vldc5s4FP01ntk+2MM3+DFfTdvZzmTXndlt3xQQoFZIrBDB3l/fKxAGDHHj1EntOjMxOpIu4tx7DkbM7KtsfStQnn7kEaYzy4jWM/t6Zlmm43vwpZCNRowgaJBEkEhjHbAi/+N2oEZLEuFiMFByTiXJh2DIGcOhHGBICF4Nh8WcDs+aowSPgFWI6Bj9h0QybdDANTr8HSZJ2p7ZNHRPhtrBGihSFPGqB9k3M/tKcC6bo2x9halir+Wlmff2kd7twgRm8ikT0H8F/7t6/+V9tl46fyUfPhkfbudOE+UB0VJf8EcSRRRXEBZwgVFUwDfjksQkRJJwVujLkZuWIxwBZbrJhUx5whmiNx16KXjJIqwWYkCrG/Mn5zmAJoBfsZQbnX9USg5QKjOqe/GayH/V9IWrW597PddrHblubNoGk2LTm6San/t93bS6NZh3hwXJsMRCgzFnUq/OVNFiQukVp1zUFNhxjL0wBLyQgn/DvZ7IX94bKgLkP1fTY4rXF6oyAat0SZnNedv6dxaeD8A4wzrpBS9FiPektVUKEgmWe8bZzTiVwN4JdP3cYg4MiE1dBxRy/zDUBNLSSrbj9FS4OrTpDcg5YbLoRb5TAAzQNjF3bK0R7RKWu1PKB06Ag2YNbat3MR1U6+MArZh7tbKjia7iVZFWKZF4laM6aRVY5bC6d4vr0cQ/YCHxem+qthQthxSZnm5XnY2ZrTelPQtrx02lt0fw4fzZI/4gjMxyFWxL5LxvNYuc5GNqWyGFSES/hls32Cm/5Zhb25jg1n8pbq3lM1x5ypGO5tRmz6c71/6hU1uG3fdq8yW82lV/U17t1Z+fcl73VznvT5WPO5JmOKeqbBlhiWKClglhM8ujcE2X9wKOEnX0x90GCoS9Oa797eQLm5GL/al8LT3fRnvzdYCofWMoasecMEz7NQ3TOzVNH/Lr68m+ezRJuSclKcs5uew9z5G9M3fk4CwdORg5cs4LOS/zCEk8L0JBcjnhx6sUYr+sHYMZB5EzlazAurf3J+sAO7b9oR27xoQdO69px2Zwaop+5m8sxxwo2lgsA+/cVN3enM9M1u2ye7pelTkWD6TgYkLPt5wilrywoBEO4sm9Cy8M8H18JEF71hME/aq/r6yTE/Qzb9GB6Z+/oM/zyckcPzrdCRU7xWUxFjTcuKF8ppT+Gz1JWcETnqRe9dZt+aMsZaV6iQAsj1KRCx7iosBq47sSwPd4A9wAAcO/tO5CGaQHOJvaqwLK5DBFI+qN+jOhtvoDOIIncQZYCClRUr1UqYC10Avd0eyePVIdQ2/aLZDtSwnVmQgUEThJuwjGmYqaty6hIqv9APv6WJXiLP2F++Nasd1xrQSH1wo0u9ctza5w99bKvvkO</diagram></mxfile>

docs/sources/middleware_overview.drawio

@@ -0,0 +1 @@
+<mxfile host="www.draw.io" modified="2019-12-06T15:59:23.910Z" agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) snap Chromium/78.0.3904.108 Chrome/78.0.3904.108 Safari/537.36" etag="M11UptGqWplE1IXWw34S" version="12.3.6" type="device" pages="1"><diagram id="0rmPf6qEohoiC0EBVp_1" name="Page-1">7Vpbc+MmFP41ntk+JCMJSY4fYyfZZGYzk6mnbbZvRMIWDRZehG/76wsCdLfjxJLtdJsHRxzgCM75zg3UA6PZ+iuD8+iRhoj0HCtc98BNz3H8gSt+JWGjCe6VIkwZDhXJzglj/BNpoqWpCxyipDSQU0o4npeJAY1jFPASDTJGV+VhE0rKb53DKaoRxgEkdepfOOSRol45/Zx+j/A0Mm+2/YHqmUEzWO8kiWBIVwUSuO2BEaOUq6fZeoSIlJ2Ri5p3t6U3WxhDMd9nAv7zgSxHq2/Pfw8ff8A/oHVzPbqwHcVmCclC71ivlm+MCBhdxCGSXKweGK4izNF4DgPZuxI6F7SIz4ho2eJxQmN+B2eYSH3fI7JEHAdQd2j12gPRri9f72iJGEfrAklv5yuiM8TZRgzRvY6vRauxdaWbq1xRrqdpUUFJjiFCDY5pxjqXn3jQInyPOL2a9FAo8KSblPGITmkMyW1OHZblm4/5RulcS/UfxPlGSw8uOC3LHK0xf5bTLz3d+q6ZyeebdbGxMY1Y7Pe52Piec5DNfFraMvMSDhm/loYlCAGBSYIDQ77D5N0wQHFouMU0RoqiGVm7YJLQBQvQLl1obwHZFPEd43w1TuppJ+gYIpDjZdkvNCEonSo2BTeFAXOKY54UOD9JQo5lUMGy7VoVNCqOOTazpR0AV/8kcD0X6DWajb3TbLqHow06wWMNcK5TBpzwimUWakd6Vo7EQ4FtvHSnwHbAKf2wXYBTDq6T+OGCe80H5R52b3sBkncE57JBcPx6HFPot20KBwV3Z3A+oHL+R9VHUeW4ZxHwPa8S8AdeOX1/a/xREoSG6sAnXKuxZAv+jwU1HRdJquBrMcAG83WqVtMvnqby/yMOQ4JWQtGGpVih4qoGHKEOAe3UIcCqqKahELGdhkLE76oOOUkZspfIXdGWkhU95JrgaSyIsxQKkhll+KcYDA3LPf3Rx52Gu28oOqtI5G41y5e9bHIgbNJR6ayV2tFFogxJ9saUzYQCagY7xHxI1wVjfcktVdEEk7i791/P54WXq3cdzVO47XgK++w8hb8VSiFe7u3ga8p6NyQauTzEHDHR+8RogBIRTK0Rnc0WsdAPxzTOQM8+/oovD0+j37biKiOnwvhsccmtos0+NdrMAg7yXE3ZxHbnNIQJqqlMCJCX9ZJwRl/RiBLKetlR1ESEmAoJ6pgVoBSbO4JZExDKUDnqWWm/jIVBAxTsprPSzqAAtkJBO4003pfQkPCi5g9IPu8hC1XqaY1RsGCYS1E90nBBJO3L/fhxu1P4rOafHXufzvz7NZ2PIzVvHDA8V/XS55Cu1z+3UO40VWvVMqCpcv+IBJuPJWy/v/tcQjSeEMNiw9J57j5SfTPZLki66ZrH0A48KM3cYnaAObj0nDKXLWeldV61w1DBq8xKlSI1Vm1V9KBugiKfCjFDgUypZOL9Zqw0MZCgCd8VARkSrhi+pIykpvWRiODqDXvejdQ9GBL4gsgQBq/T1NRNuO05YJL+VbC3M27WobTbUKrGnl0j60X3ije1TU7AunRc128FZuZ0cWM4V3BBJ5MEdQIJ4ybe7Te2VOetOuRzdQJ+zXA/7APqrLr2AYOawl9+NRdgMN+CC3DdAWjHBYAjGv32M6SDCn9RqoulRWiRfPbquZbgnbx6Nh8dnejrkt7+t5rvv0dq/Phj/2zU3Hdl3xeoVfbfvPBq6/bJ3J4f/yS5ldunyq382+MdrwLnLm7xvW581O8oxP8593T66h40qaur+nNfwz3T/BFYreWPDay6zh/rlwY/Fkhu/w6tUfALpI8Z1NuoIH0AypbcTjJ5ZVBRZntIaima+ffKanj+0Te4/Rc=</diagram></mxfile>