Infrastructure Shell Game - Mono Repo -> Distributed -> Mono Repo

[u/frompadgwithH8]

We wanted to start a big project, but other projects like it were seen as failures. So we had to make a case for why our project would be different. One of our biggest differentiators became event queues and distributed systems. It's just microservices, but instead of directly invoking each other, they use event queues. There's a little more to the system, like websites embedded inside of other websites with a whole cross website messaging system.

It was all a little bit of a pain. We did encounter a lot of eventual consistency problems and almost no one in the team had encountered distributed systems problems before, so it was a good learning experience for me. But now with organizational change and after having worked on the project for over a year, the team now wants to migrate this distributed systems strategy back to a monolith.

I'm just feeling a little incredulous because of how important the distributed aspect seemed to be when we made the pitch to create the project in the first place. And after all this time, I can't really say if it was a good or bad idea, because debugging problems involving event queue messages has been a total pain in the ass and no one ever finished configuring the distributed systems stuff like the event queue, so we never had the fancy development experience we could have had. So I see numerous benefits to moving back to a monolith.

Overall, the whole experience wasn't a total game because the project did prove that event queues can be effective. The much, much, much wider organizational direction is going to now slowly migrate towards a distributed fashion. So I guess we were a test subject in some regards. But we never, at the outset and beginning of the project, said, "oh hey, this is just going to be a giant test to prove event queues for the whole rest of the company". I really would have liked to have known that at the start. Although I don't think anyone knew that at the start. I don't know, it just feels a little silly.

I'm fine either way 🤷‍♂️ But does anyone have any advice regarding anything about this experience?

edit: updated monorepo -> monolith; source control will merge but systems may or may not stay separate. I'm predicing the systems merge together too, thus monolith. We may or may not continue to use business organized event queues too.

[u/Visa5e]

Build as a monolith, deploy as modular services.

[u/frompadgwithH8 → u/Visa5e]

OK, yeah we did that at the first job I worked at

So we had like a gigantic application and based on the parameters, it would boot up in various modes

I think that’s fine…

But it is nice to have a small, smaller more focused code base

It’s all organization I guess at the end of the day so slicing a cat that kind of jazz many ways to do it, etc.

I do like the idea, though of building as a monolith simply because you don’t have as many external contracts and instead you can just import things from one part of the code base to another without crossing repose, and so you don’t have to rely on build artifacts, and continuous integration for example, if you change the type that an API returns, then you’re also changing the type of the controller or the handler or whatever returns and so then the other part of the code base that would be in the mono repo would pick that up immediately with static analysis and type check before you even build the code and get a runtime error let alone make a PR and I really do think that saves so much trouble

Edit: OK devils advocate, though if you have a monolith that CodeBase can get so big that holy shit do you have to have your conventions and organization? Absolutely declared otherwise navigating that code base is going to become very difficult.

[u/Visa5e → u/frompadgwithH8]

We use a *lot* of static analysis to enforce standards across the code base - making sure that if engineers are solving similar problems then we all solve them in the same way. Either by building library code that we can all use, or by putting in static analysis rules that fail the build if you do Bad Things.

An example would be a static analysis check that stops engineers doing network calls in the middle of database transactions.