[FEATURE]: Monorepo support

[cgorman]

Feature Request

Hi there, I was going to add this as a comment to #1477 but I'm not positive that they are related so I figured I'd add it as a separate request. I think, more or less, this would ask that we can have multiple garden projects in a single repository, and that projects can reference modules which are not subdirectories of the projects. I'm going to go into maybe too much detail to make sure there's no crossed wires!

For example, given this structure:

FakeMonoRepo/
├── README.md
├── docs/
└── src/
    ├── some_web_service/
    │   ├── Dockerfile
    │   └── src/
    ├── frontend_service/
    │   ├── Dockerfile
    │   └── src/
    ├── python_package_project/
    │   ├── Dockerfile
    │   └── src/
    ├── python_web_service/
    │   ├── Dockerfile
    │   └── src/
    └── some_app/
        └── k8s/
            └── app.yaml

I'd want to be able to make some_app have a Project definition which references Module definitions in python_web_service and frontend_service or something like that.

Background / Motivation

I honestly don't know if we are just organizing our repo in a terrible way because nothing seems to support it, but here's how we do it right now:

• Each subdirectory of src is considered a "project". I'm putting our idea of "project" in quotes to avoid confusing it with the Garden idea of a project.
• A "project" can be a library/package, command-line application, web service, or Kubernetes application
  • In the previous example, some_app would be a Kubernetes application which uses containers defined in frontend_service and python_web_service
• Each "project" has a Dockerfile
  • Each Dockerfile defines at least two targets, shared (or something like that) and test.
    • One target builds the container image and the application, another target installs test dependencies and executes the unit tests (e.g. by setting a test.sh script as the entrypoint)
    • If the "project" is a containerized deployment, like a web service or a containerized CLI application, then the Dockerfile will have another target for the application

We want to have a consistent process across all of the "projects" in the monorepo. We especially want to be able to do this because in the future we will be providing applications which are just Argo workflows, and multiple workflows will reference the same container images.

What should the user be able to do?

I think I'd like to be able to have multiple garden Projects as subdirectories in a git repository, and I'd like multiple Projects to reference the same Modules. Modules and Projects are at the same directory level.

As I'm writing this I realize that the way we run tests in Docker might not be the best way to do it moving forward so I'm sure we can change that if need be. The only issue I can picture right now is that we don't want the deployment image to have testing software and stuff included.

Why do they want to do this? What problem does it solve?

It would allow us to use Garden, and so far Garden looks like the only tool that does anything close to what we want a tool like it to do.

Suggested Implementation(s)

I honestly don't know, but naively I'd say that maybe having some kind of concept of a subproject, or maybe make the root config be a Repository and then each Repository can contain Modules and Projects? Importantly, each Project still has its own stack graph. I don't know enough about Garden's internals so I can't really speak too intelligently about this.

How important is this feature for you/your team?

🥀 Crucial, Garden is unusable for us without it

Hope this makes sense, thanks!

[sra]

We have similar setup where we use a monorepo and have different directories holding groups of related services. There is also a helm chart there that packages stuff up for k8s deployment. I've not had great luck finding a way to get garden.io to work.

[eysi09]

Hi @cgorman and @sra!

Allowing smoother coupling of projects is definitely on our roadmap. For example via project inputs and project dependencies as mentioned in the GitHub issue you referenced. This will allow user to compose projects into even larger ones.

That being said, it looks what you're describing should be solved by just using Modules. So some_web_service, python_web_service, etc should be modules in the overall project, even though they could conceptually be considered projects to the team that's working on them. Note that you can also disable modules in a given project, for example based on environment or other variables. Here's our introduction guide to modules.

This how we structure our own projects and what most our users do as well.

As for code sharing between modules, that's e.g. possible by using build dependencies. Here's one example of that.

All that being said, there are currently some "inconveniences" with respect to code sharing. In fact, I just opened #1954 to mitigate those. The issue also discusses the currently recommended workarounds, such as using build dependencies as mentioned above. This is also related #853, the comments there provide more background context.

Does this help your respective use cases?

[eysi09]

FYI: I also added a short entry on this in our FAQ (#1955) with the note that we should write a proper guide on this when we merge #1954.

[cgorman]

Hi @eysi09

With this setup I don't think we'd be able to use modules in multiple applications, right? Here's a (hopefully understandable) semi-concrete example (not using the structure from my original post, sorry):

Application_1 gets input data from a user and applies some transformations and such, then posts the result to AppServer_A.
Application_2 gets some other input data from files, as well as from AppServer_A, and applies some more transformations, then posts the results to AppServer_A and AppServer_B. In our repository, Application_1, Application_2, AppServer_A, and AppServer_B are all at the same level, e.g. within MonoRepo/src/.

We'd like to develop, test, and deploy Application_1 and Application_2 separately even though they both depend on the service provided by AppServer_A. The two Applications are independent from each other. If we weren't using a monorepo I imagine we'd have separate git repositories for each of these and register the AppServers as garden modules and the Applications as garden projects. In production there would only be one instance of each AppServer running as well. Let me know if you need any further clarification, thanks!

[stale]

This issue has been automatically marked as stale because it hasn't had any activity in 60 days. It will be closed in 14 days if no further activity occurs (e.g. changing labels, comments, commits, etc.). Please feel free to tag a maintainer and ask them to remove the label if you think it doesn't apply. Thank you for submitting this issue and helping make Garden a better product!