[discussion] engine rearchitecting

[glennfu]

@defkev brought up some interesting points recently about potential changes to the engine.js. I'd like to start a larger discussion about some bigger goals for the engine in hopes that if a comprehensive overhaul is to happen, it can encompass some extra ideas.

Goal 1: I want it to be possible to run multiple coins. Right now I can set --buy_max_amt and run 2 separate bots, and with a strategy configured to switch modes based on whether or not I have a positive asset balance, they do indeed play nicely together without overspending beyond that buy_max_amt. However, because they, on Binance at least, are running 2 separate polls to Binance, I get my IP throttled when I try. You can implement websockets for multiple coins at 1 time, but how do you get 2 instances of Zenbot to share that stream of trades? Definitely need a major change for that.

Goal 2: Fix the madness of this setTimeout chain. It's entirely possible (and happens to me on occasion) for an error to trigger a retry of the callback chain of an order, and then in the next period, an entirely separate chain of callbacks to trigger on the same order. There's no way for the engine to recognize that it's already looping through a bunch of setTimeouts and so they start stepping on each others' toes. The best case scenario is they both try to place an order, and the exchange sends an "insufficient funds" error causing 1 chain to die. Timing should somehow be more explicit. Either decisions should be made as callbacks from web socket streams, or after time passes, it should be saying "with the current state of things, what should I be doing with my current order?" instead of hoping that the setTimeout dance in the ether will figure itself out. There's multiple related bugs that this could solve.

[defkev]

After #1359 came up i was thinking about using nodes cluster feature.
https://nodejs.org/api/cluster.html

Having one master process running the "core" stuff like talking with the exchange(s) and N workers doing the actual strategy/trades.
That way all workers would've access to the same set of data, taking out a lot of overhead multiple running instances create, including hammering the exchange.

This would also allow us to expose multiple instances under a single IP/port for the API, along with something like a "Master" dashboard listing all running instances (with some brief info like exchange/selector/profit) will clicking on one brings you to the instance dashboard, as we have it right now.

This should at least partially solve Goal 1

Definitely would be a bigger change non the less, as we most likely need to touch everything under the hood.

Maybe with v5 😁

[carlos8f]

Hey guys, cross post, Bot18 is my new bot in development and has already solved these issues! Will probably drop soon! Stay tuned!

#1589

~Carlos