MPP-3079: Add a C4 Relay model, initial diagrams

[jwhitlock]

• Add the 2 top-level C4 models for the system context and containers.
• Add a dynamic view for happy-path email forwarding (new implementation of this doc)
• Add a deployment models for dev, stage, and prod
• Use structurizr lite to layout, create diagrams
• Add docs/system_diagrams.md to document the diagrams

How to test:

• Read docs/system_diagrams.md
• Try strucurizr lite, play the dynamic diagram

[groovecoder]

This is very thorough work. Thank you for going thru it all.

I made it thru the C1 level before I was overwhelmed by the jump in the level of detail at the C2 level. I can go thru the full diagram, but I thought I'd give a quick review with some comments that might help to simplify the C2 level.

1. At C2 level (maybe even at the C1 level), we should show Amazon and Twilio as external software systems that interact with Firefox Relay, Phone Contact, and Email Contact.
2. At C2, we should consolidate the individual AWS elements together into something like a single Email Processing container. The details about Sender, Receiver, Storage, Queues, Pub/Sub, and Encryption are good though, so maybe pull them out into their own C3 diagram that details the Email Processing container.
   • These details are also great for the dynamic diagram for forwarding an email to a Relay user!
   • When we have an "Email Processing" C3 diagram, we can similarly consolidate the deployment diagrams to replace the duplicated details in each deployment into a single "Email Processing" system.
3. Similarly, we should group the individual Tasks into something like a Python/Django Commands container. And then move the invididual task details to their own C3 diagram.
   • And then similarly, when we have a Python/Django Commands C3 diagram, we can replace the duplicated details in each deployment into a single "Python/Django Commands" system.

[jwhitlock]

I've made the smaller changes, but the bigger ones will take some time, so it will be a bit later before this is ready for re-review. With PTO and MozWeek, it will be August 22 at the earliest. If the delay concerns you, please consider merging this work and reviewing the next PR later this month.

I made it thru the C1 level before I was overwhelmed by the jump in the level of detail at the C2 level. I can go thru the full diagram, but I thought I'd give a quick review with some comments that might help to simplify the C2 level.

I've been chewing on this, and you're right, the jump is too much. It was important for going to the deployment diagrams, which might deserve that detail level for completeness, but it does not help understanding. It's addressed on the C4 model website:

How do you diagram large and complex software systems?

Even with a relatively small software system, it's tempting to try and include the entire story on a single diagram. For example, if you have a web application, it seems logical to create a single component diagram that shows all of the components that make up that web application. Unless your software system really is that small, you're likely to run out of room on the diagram canvas or find it difficult to discover a layout that isn't cluttered by a myriad of overlapping lines. Using a larger diagram canvas can sometimes help, but large diagrams are usually hard to interpret and comprehend because the cognitive load is too high. And if nobody understands the diagram, nobody is going to look at it.

Instead, don't be afraid to split that single complex diagram into a larger number of simpler diagrams, each with a specific focus around a business area, functional area, functional grouping, bounded context, use case, user interaction, feature set, etc. The key is to ensure that each of the separate diagrams tells a different part of the same overall story, at the same level of abstraction. You can also use an alternative visualisation.

I have some ideas for that, and added some TODOs to system_diagrams.md. Until then, I've labelled this the "(All Details)" variant of the diagram.

At C2 level (maybe even at the C1 level), we should show Amazon and Twilio as external software systems that interact with Firefox Relay, Phone Contact, and Email Contact.

That was in my original plan:

However, Structurizr would not show any second level relationships for the System Context (C1) - it showed the phone system but not the external contact.

Going back to the C4 model website

Should data storage services be shown as software systems or containers?

A frequently asked question is whether services like Amazon S3, Amazon RDS, Azure SQL Database, content delivery networks, etc should be shown as software systems or containers. After all, these are external services that most of us don't own or run ourselves.

If you're building a software system that is using Amazon S3 for storing data, it's true that you don't run S3 yourself, but you do have ownership and responsibility for the buckets you are using. Similarly with Amazon RDS, you have (more or less) complete control over any database schemas that you create. For this reason, treat them as containers because they are an integral part of your software architecture, although they are hosted elsewhere.

Once I treated email processing as several containers, and Twilio as "our" phone service, it felt like I was working with Structurizr rather than against it.

• The External Contacts came back at the System Context (C1) level. It then made sense to distinguish between email and phone contacts
• It became trivial to show the alternate iQ provider for development
• The dynamic diagram for forwarding an email was easy

There is still grouping in the deployment diagrams to show GCP vs AWS, and the different services used in each.

After working with it a bit, I think AWS and Twilio belong inside the Relay system container, not external.

At C2, we should consolidate the individual AWS elements together into something like a single Email Processing container. The details about Sender, Receiver, Storage, Queues, Pub/Sub, and Encryption are good though, so maybe pull them out into their own C3 diagram that details the Email Processing container.

These details are also great for the dynamic diagram for forwarding an email to a Relay user!

When we have an "Email Processing" C3 diagram, we can similarly consolidate the deployment diagrams to replace the duplicated details in each deployment into a single "Email Processing" system.

As I said above, my plan is now to create a summary container diagram that treats all of the AWS parts as one "box", as well as collects other parts in their own boxes, to make a diagram that is about the complexity level of the system context diagram (C1), and could potentially fit on a slide. Then, there will be other C2 diagrams that "zoom in" and show those systems. The dynamic diagram seems the right size for the zoomed-in email processing piece. I think the other two zoomed-in diagrams are clients and the web application, and the scheduled tasks. If those are two crowded, there may be one for the web app and the backing services. The current noisy one stays on as a detailed view, and a transition to the deployment diagrams. Those may get broken into summary and detail as well.

I think the hexagon would make a decent shape for "this element represents a collection of elements"

Similarly, we should group the individual Tasks into something like a Python/Django Commands container. And then move the invididual task details to their own C3 diagram.
And then similarly, when we have a Python/Django Commands C3 diagram, we can replace the duplicated details in each deployment into a single "Python/Django Commands" system.

I disagree with treating the tasks as C3-level components. The C2-level container is "a separately runnable/deployable unit (e.g. a separate process space) that executes code or stores data". The C3-level component is for the software architecture. I think the separate C2-level diagrams will get to the same results of understandable diagrams.

I was going to wait until a future PR to break the web application into a C3 components diagram, but it may make sense to do it now to test how things will look at that level. I think a summary and details version may make sense for that as well.

[groovecoder]

Thanks for the thoughtful reply.

I think the separate C2-level diagrams will get to the same results of understandable diagrams.

Yeah that sounds just as good and more aligned with C4 & Structurizr like you said.

If the delay concerns you, please consider merging this work and reviewing the next PR later this month.

I always get concerned if we leave things un-merged for too long. I'll go ahead and merge this one and make a Jira ticket to make follow-up changes.

[groovecoder]

I filed https://mozilla-hub.atlassian.net/browse/MPP-3864 for the follow-up.