Designdecisions regarding on-prem such as network exchange format, interaction layer, integrations

[GGAlanSmithee]

As came up in offline conversation with @brokenprogrammer, there are some questions that are not answered in regards to version 1 of the feldmanchain-on-prem client. This results in me making a lot of assumptions regarding how the client should be interacted with, what network exchange format we should use etc. Let's use this issue to discuss such things, and potentially move the discussions to their own issues when warranted.

Things that has come up:

• network exchange format, should we use binary, JSON, protobuf etc? Pros and cons
• Major way of interaction. Should we build a github integration, github hooks, a RESTful web api?
• more?

Thanks for bringing this up @brokenprogrammer

[brokenprogrammer]

network exchange format, should we use binary, JSON, protobuf etc? Pros and cons

I strongly believe we should disregard this as it is a much simpler problem to solve than the project as a whole. For now this is unimportant and we should just use what gets us forward the fastest.. Plain text is not out of the question but JSON is standard in JS which we're currently using so why not stick with it?

Major way of interaction. Should we build a github integration, github hooks, a RESTful web api?

As I understand it, me as a project owner should have a client running which dispatches jobs if there are any. How does it know if there are any jobs?

Either you tell it manually (No thanks, I want to do less work not more) or you get someone else to tell it for you. In this case it can be Github, Azure devops or even git itself. I think this part we should keep kinda open in the sense that perhaps developers can make their own integration here somehow.. But that is out of scope for now.

The three mentioned (github integration, github hooks and rest api) are all the same. Could we turn the client into both an web server and a p2p peer at the same time? In that case we would have a web interface for services such as git to communicate that a build is ready and the rest solves itself kinda?

[GGAlanSmithee]

Could we turn the client into both an web server and a p2p peer at the same time? In that case we would have a web interface for services such as git to communicate that a build is ready and the rest solves itself kinda?

This is exactly what I am thinking. This way we separate the invoking from the execution.

[GGAlanSmithee]

To expand on my last post, I was planning to add a RESTful web API to first the project author client, and eventually the builder client as well. The idea is that it can be run locally and is agnostic to who- or whatever integrates with it.

For the near future this will make it easy to setup a PoC using githooks for example, which I think will be the lowest hanging fruit. I am also planning a simple SPA UI using the API as a backend for testing purposes. Further down the road, things such as API security can be handled by the cryptograhic keys already used by the client for the peer-to-peer communication since it is intended to be run locally.

I will submit a PR with a thin RESTful API in the not to distant future.

[GGAlanSmithee]

I think it actually makes sense to merge (or remove tge concept of) the clients and add some feature flag detection logic to allow a client to be one of or both of a client and a builder (and in the future any other entity such as an account, an auctioneer, an excrow intermediary etc)

[brokenprogrammer]

But you can't host a rest api locally on your machine if you are running for example github webhooks you need a server for that so it still makes sense for the client to be an actually hosted server that you have to host on the web while the builders are simply listening to messages through p2p from those servers.

[GGAlanSmithee]

Yeah sure, you can host it however you want - in the case of using GitHub hooks ofc it needs to be publicly available. Personally, I am thinking about using Git hooks for the first version, but the client itself will be agnostic to how is called, so anything works.

I think it makes sense to not trouble ourselves with the method of invocation for the first version, since the technical challenges lies elsewhere.

[GGAlanSmithee]

Also, it somehow plays in to the on-prem theme to not involve external (to the local net) repository solutions such as GitHub. This usecase will make more sense in the global version imo.

One setup which I do think makes sense to support in this context is to have a server with a git remote on it, setup git hooks on it and have feldmanchain orchestrate builds on pushes to it.

What do you think?

[brokenprogrammer]

If we rely on git hooks who keeps track of builds and stores them and gives me as a user the information I want? You might be able to hack something together with server side hooks but I don't know how flexible it is.

I feel it is still vailid to host a server to handle on-prem since if your git server is hosted locally it would just simple work as well. For example if you would set up gitea then it supports webhooks as well which I am guesing would work locally.

[GGAlanSmithee]

Yes, sure, what I am getting at it is that it shouldn't really matter how you call the client. As far as the client itself is concerned it does not. If someone wants to use Github hooks, then that user can do that, but to the extent that we need to go out of our way to support github, I would say that is out of scope for sure, since that would be making a lot of assumptions and quite weird within in the context of a p2p builder network, do you agree?

[GGAlanSmithee]

If we rely on git hooks

Sorry if I was unclear, but I do not suggest that we rely on any specific way of integration, but rather decouple the solution from any specific method of usage. Atleast until we have some form of technical solution to the core problem we are trying to solve. As a PoC any method of invocation should do.

[brokenprogrammer]

I am just trying to see what path we are taking moving forward to get closer to a working PoC. Imo the path with most unknowns is a restful service that I can host which accepts a http request from a webhook. I do not understand the other solution you are proposing.

[GGAlanSmithee]

a restful service that I can host which accepts a http request

This is what I am proposing as well. Have already implemented this, but need to finalize it, expect a PR soonish!

[brokenprogrammer]

Interesting, so me as an end user will be able to customize this in order to get it working with for example my github projects? If that is the case It sounds really good.

[GGAlanSmithee]

@brokenprogrammer most def! And if somehow, I've missunderstood anything, we will just change it, but that is absolutley the goal

[GGAlanSmithee]

@brokenprogrammer I've started implementing an API now, but we need to discuss how to move forward from there. We could use the Github integration as a usecase to have something concrete to anchor the implementation on.

[brokenprogrammer]

My personal usecase is a job triggered by webhook (API Request) and then then the builder just runs my build.bat, I think for a first usecase we could also do like an npm build since its the same simplicity and more close to what you've built?

[GGAlanSmithee]

Sounds good!

What would an API endpoint that's compatible with the webhook look like? Is this Github specific, or some third-party CI? Is there any documentation available?

I agree than a npm build certainly is reasonable for a couple of reasons;

1. All dependencies are publically available at the npm registry (not taking private/local registers into account)
2. It is self-contained such that all dependencies, once installed, are kept locally in the node_modules folder
3. NodeJS / JS has great cross-platform support
4. Specification of both dependencies as well as constraints (such as NodeJS engine levels) are co-located in package.json
5. There are a lot of NodeJS projects availalbe here at Github, that can be used to test with

So yeah, if we use a NodeJS project as a primary usecase, considering all of the above-mentioned points, it would probably provide the best conditions for an actual builder as well as for the feldmanchain as an architecture, with regards to communication. I think a second use case could be a c build, using firstly a straight forward bat script as you suggest, and maybe secondly a make based project.

[brokenprogrammer]

I think the API endpoint would unfortunately have to be custom depending on the integration, for github specifically they have this documentation: https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads

However porting this solution would basically only result in a different payload so perhaps some sort of generic solution can be made there in the future.

[GGAlanSmithee]

However porting this solution would basically only result in a different payload so perhaps some sort of generic solution can be made there in the future.

Yes, either this or providing the functionality to add custom endpoints. Thanks for the info, will think on it a bit

[GGAlanSmithee]

@brokenprogrammer

With the latest PR getting merged I think we are at a place where we could start thinking about:

1. How to call into the feldmanchain (be it via Github actions or w/e) which is possible now that we have an API.
2. How to actually request, perform and receive a build

Regarding point 1, as previously discussed, there exists an API that, by default, is served on localhost on a given port. How and if to expose that port publically is up to the user for now. So either you call into it locally, or you expose it publically and call into it from for example github actions.

Exactly what API routes we need to add and what the should do is something we need to discuss.

Regarding point 2, that also requires some discussion since it is not clear how that should work in relation to for example github actions and/or the feldmanchain builder.

Any initial thoughts?

[brokenprogrammer]

How to call into the feldmanchain (be it via Github actions or w/e) which is possible now that we have an API.

I think as a startingpoint we could add a github webhook on this repository that sends a request to a running version of the api. We can use ngrok for testing.

How to actually request, perform and receive a build

Build is initially requested from Github, the API recieves a message on the route we registered for the webhook. Sends a build request which could be literally anything for now and then performing the build could for example be running npm build on the builder side of things. We can later change it to run an application that we made.

[GGAlanSmithee]

Sounds good, I'll start with this as soon as I get the time.

[GGAlanSmithee]

We've been a bit lowkey on this project over the hollidays, but talking about it in person with @brokenprogrammer we are going to move forward with the on-prem version to try and get a basic builder ASAP.

To recap on that conversation, we are talking about the seperation of concerns in regards to the builder, the API for the builder and the p2p network (AKA the blockchain) as well as the various integrations such as Github actions, webhooks etc.

As it stands now, @brokenprogrammer are looking to create the most basic integration between the current chain and builder while keeping the seperated and not making too many assumptions at this early stage. Please add any additional information @brokenprogrammer