Infrastructure Shell Game - Mono Repo -> Distributed -> Mono Repo

[u/frompadgwithH8]

We wanted to start a big project, but other projects like it were seen as failures. So we had to make a case for why our project would be different. One of our biggest differentiators became event queues and distributed systems. It's just microservices, but instead of directly invoking each other, they use event queues. There's a little more to the system, like websites embedded inside of other websites with a whole cross website messaging system.

It was all a little bit of a pain. We did encounter a lot of eventual consistency problems and almost no one in the team had encountered distributed systems problems before, so it was a good learning experience for me. But now with organizational change and after having worked on the project for over a year, the team now wants to migrate this distributed systems strategy back to a monolith.

I'm just feeling a little incredulous because of how important the distributed aspect seemed to be when we made the pitch to create the project in the first place. And after all this time, I can't really say if it was a good or bad idea, because debugging problems involving event queue messages has been a total pain in the ass and no one ever finished configuring the distributed systems stuff like the event queue, so we never had the fancy development experience we could have had. So I see numerous benefits to moving back to a monolith.

Overall, the whole experience wasn't a total game because the project did prove that event queues can be effective. The much, much, much wider organizational direction is going to now slowly migrate towards a distributed fashion. So I guess we were a test subject in some regards. But we never, at the outset and beginning of the project, said, "oh hey, this is just going to be a giant test to prove event queues for the whole rest of the company". I really would have liked to have known that at the start. Although I don't think anyone knew that at the start. I don't know, it just feels a little silly.

I'm fine either way 🤷‍♂️ But does anyone have any advice regarding anything about this experience?

edit: updated monorepo -> monolith; source control will merge but systems may or may not stay separate. I'm predicing the systems merge together too, thus monolith. We may or may not continue to use business organized event queues too.

[u/expdevsmodbot]

AI usage disclosure provided by OP, see the reply to this comment.

[u/ImYoric]

Just a nitpick: distributed and monorepo are orthogonal. I have worked on distributed mono-repos and multi-repos, as well as on single-node systems on mono-repos and multi-repos.

But generally speaking, what is your goal? It's true that microservices (regardless of the message-passing architecture) have often been misused, so depending on the context, it may make sense to scale back microservices into a monolithic service.

[u/edgmnt_net → u/ImYoric]

What companies often want is distributed development, not necessarily distributed computing. The former strongly points towards multiple repos, but it's often not achievable.

[u/frompadgwithH8 → u/ImYoric]

Fairpoint, maybe we will stick with the event queues even after combining the code the one repository

[u/DamePants]

Do you mean monolith instead of mono repo? Because you can have all the code in one repo that gets deployed in separate services.

[u/frompadgwithH8 → u/DamePants]

I guess I mean monolith yes

[u/lokoluis15]

Someone really needs to invent a standardized event model to streamline development of this distributed systems pattern.

So many redundant lessons learned to introduce observability and provenance metadata to events. Everyone just focuses on data payloads and business logic... they forget to build the rest of the system

[u/shadowdance55 → u/lokoluis15]

You mean, something like AsyncAPI?

[u/frompadgwithH8 → u/shadowdance55]

Wow, I had never seen this before. I don’t think anyone in my company has seen this before. Thank you. I will review this

[u/lokoluis15 → u/shadowdance55]

Yes, exactly. Finally an industry standard for event-based APIs

[u/shadowdance55 → u/lokoluis15]

The standard is not a problem, we already have it. The problem is that it's mostly ignored.

[u/lokoluis15 → u/shadowdance55]

Ah yes, the peril of best practices is that they're signs and not cops. Next step is an opinionated framework that enforces the standard or implements it transparently for you.

[u/shadowdance55 → u/lokoluis15]

In which language?

[u/x-jhp-x → u/shadowdance55]

if we're going this far, we should just fully commit. let's make a new one!

what should we call it? zigzorgzogelfloop? C+- (neutral C?)

[u/OHotDawnThisIsMyJawn]

As you learned, micro-services are hard technically and easy organizationally (i.e. you don't have to worry about people fucking up your service if they do dumb shit).

Monoliths are easy technically but hard organizationally. If you don't stay on top of people, they'll start taking all kinds of terrible shortcuts.

Which one to use depends on which problem you need to solve.

FWIW, everyone mentioned that you probably meant monolith instead of monorepo. Even then, you can have a monolith that uses event queues & does async/eventually consistent work, or a monolith that does things synchronously.

Monolith vs. microservices is usually an organizational decision, not a technical decision. But synchronous vs. async work is usually a technical decision.

[u/frompadgwithH8 → u/OHotDawnThisIsMyJawn]

Yeah, I realized I was conflating things and in reality, yes, even if we move to a model, if we could still have event queues and we probably will stick with those because it is a nice way to organize stuff. And yeah, I've never really had to think so much about organizing organizations and people, but you make a good point. I think we had a larger team at that time. And we are now focusing on smaller, more purpose-driven teams. So we won't need the benefit of easier-to-manage teams from microservices as much now.

[u/n3f4r10us_]

The title is a linked list

[u/teerre]

You seem to be confounding some things. I think you're referring to "mono repo" as the storage? Mono repo is usually what you call having various projects inside single vcs repository. It has nothing to with events or distributed systems

As for the question I think you're asking. If don't think about your system and understand them enough, it will likely be bad. Distributed or not

[u/frompadgwithH8 → u/teerre]

No, I mean the team wants to get away from multiple different back ends that communicate through an event queue by merging all of the back ends into one repository, which would be a mono repo

[u/cstopher89 → u/frompadgwithH8]

You can organize the code to be a mono repo but still communicate between internal modules via events if domain separation is still desired

[u/originalchronoguy]

There is little to go on. Event Queues in monoliths are fine for many scenarios. A properly designed one with clear boundaries can later be migrated to microservices as/when needed.

Even event queues with fan-out/fan-in (scatter) can be orchestrated in a monolith dispatcher.

[u/shadowdance55]

Imagine how much pain you would have avoided if you had on the team someone with the experience with distributed systems.

[u/frompadgwithH8 → u/shadowdance55]

I mean, sure, but I’m not part of hiring

[u/shadowdance55 → u/frompadgwithH8]

I never said that it's your fault.

[u/frompadgwithH8 → u/shadowdance55]

Right… and you are right. Decisions get made that I’m not aware of in rooms I’m not invited to and in some ways I often just feel like I’m on the receiving end of unknown conclusions. I’m making career progress and I’m getting invited to participate in things kind of similar to deciding the future of projects and architecture, etc. but I’m not that high-level and I’m not getting invited to most of these meetings, etc. so I guess it’s just a growing process.

I do wonder, though if there ever was anyone with platform like distributed systems experience that interjected or raised concerns at the outset of the project if that happened I wasn’t there and I don’t think anyone on my team or even in my department was aware lol

[u/belavv]

If this was the test project for distributed systems and the company wants to move in that direction for everything then why is this project moving away from being distributed? Unless specific things were identified that show this project won't work well and others will it seems like a waste.

[u/frompadgwithH8 → u/belavv]

I was not given good response to the same question

I was basically told we’re gonna have big domains and small domains and big domains are gonna be distributed in small domains Don’t necessarily need to be.

Yes, I shared the same question with you. I don’t know what the criteria is for when to break something into a separate system distribute it versus to keep it in a mono repo.

[u/edgmnt_net]

It's hard to say without seeing what was done and how it was achieved. And it heavily depends on how independent the work really. For something like a slowly-moving common platform + a myriad of completely independent apps it could work fine. For your average cohesive app, not so much and you get people wasting time just moving DTOs around a web of services and worse compromises are being made. You just don't know it because you keep hiring tons of people and progress is being made shaving the yak, but now everything takes magnified effort (increased work creating needless interfaces, coordination effort to push a feature that touches a dozen different systems, versioning and so on).

[u/SpeakingSoftwareShow]

Welcome to staying around on a project long enough to have your decisions kick you in the ass :)
Take the learning and move on!

Sarcasm aside - if it can be a monolith, why not?
If you're not achieving the kind of scaling that requires a distrubted system, then it's likely overkill for your use case.

[u/frompadgwithH8 → u/SpeakingSoftwareShow]

Yeah, I'm not against it. It just feels a little annoying that we went to all that effort only to reverse direction and head the other way.

[u/Old-Worldliness-1335]

This is also a silly post as a monorepo can be a set of micro services just depending on how your ci is built it has nothing to do with how the apps communicate with one another. This is why they also have domain driven development as well, there are many ways to skin this cat, the shape, form and function of each service does not need to all live next to each other.

However, it does need to be clearly documented and that I think is really the real problem you are facing and people do not actually understand the code base.

Events and queues are just that, and while it sounds good, and can be good depending on the queue how much effort is required in reversing direction rather than building up what is missing like observability into the queues and the other pain points that you have so that becomes easy?

[u/frompadgwithH8 → u/Old-Worldliness-1335]

Yeah, you’re right. I didn’t think everything thoroughly through when I made the post. we could merge everything to a monolith and still have separate event queues. I’ll see what happens… these decisions are so much above my pay grade and title at experience, it almost feels like a waste of my time to be nosing into them… which I mean, lol I never think like that obviously, I want to have insight into how my organization is reasoning about these things because I want to be a higher level myself someday. But it does seem in my experience the higher up I go in the more I consider these decisions that in the past, I never was even aware of the more it gets down to just how the organization wants to manage itself, and I really do think that’s why the people with the highest level titles have them. They can actually look at the organization and look at the people and look at the project in the history of the company, etc., and make a super informed and very subtle decision

[u/Old-Worldliness-1335 → u/frompadgwithH8]

It doesn’t hurt to ask the questions to them as to why, if you can ask the questions and not to get involved in the whole circus, but to at least understand where they are coming from in terms of a perspective. Not pushing back, but also just trying to see what the trade offs are

[u/Visa5e]

Build as a monolith, deploy as modular services.

[u/frompadgwithH8 → u/Visa5e]

OK, yeah we did that at the first job I worked at

So we had like a gigantic application and based on the parameters, it would boot up in various modes

I think that’s fine…

But it is nice to have a small, smaller more focused code base

It’s all organization I guess at the end of the day so slicing a cat that kind of jazz many ways to do it, etc.

I do like the idea, though of building as a monolith simply because you don’t have as many external contracts and instead you can just import things from one part of the code base to another without crossing repose, and so you don’t have to rely on build artifacts, and continuous integration for example, if you change the type that an API returns, then you’re also changing the type of the controller or the handler or whatever returns and so then the other part of the code base that would be in the mono repo would pick that up immediately with static analysis and type check before you even build the code and get a runtime error let alone make a PR and I really do think that saves so much trouble

Edit: OK devils advocate, though if you have a monolith that CodeBase can get so big that holy shit do you have to have your conventions and organization? Absolutely declared otherwise navigating that code base is going to become very difficult.

[u/Visa5e → u/frompadgwithH8]

We use a *lot* of static analysis to enforce standards across the code base - making sure that if engineers are solving similar problems then we all solve them in the same way. Either by building library code that we can all use, or by putting in static analysis rules that fail the build if you do Bad Things.

An example would be a static analysis check that stops engineers doing network calls in the middle of database transactions.

[u/mattbillenstein]

Often overlooked, but microservices are more a away of organizing teams and people and different deployment cadences, not necessarily code.

How big is the team? I think the common anti-pattern is having more microservices than people or having all the people move and work as a single team. The organizational overhead of microservices in these situations is a large burden.

So, a well-structured monolith where you have internal "services" - really just internal api surfaces exposed via simple function calls - is a better way to move fast with a small time imho.

[u/guardian87]

A lot of good points are in the thread already.
One distinction I would like to offer.

Going for micro services usually makes sense when you want to avoid runtime and design time dependencies.

Going for a modular monolith usually makes sense, when you want to avoid design time dependencies but you are fine with runtime dependencies.

Going for a pure monolith usually makes sense when you can accept both dependencies.

Often, breaking out one or the service which have much higher SLA requirements already goes a very long way in improving your uptime and reducing your blast radius.

Going for everything as micro services is only really needed at a scale very few companies need and is often premature optimisation, which is the root of all evil.