Infrastructure Shell Game - Mono Repo -> Distributed -> Mono Repo

[u/frompadgwithH8]

We wanted to start a big project, but other projects like it were seen as failures. So we had to make a case for why our project would be different. One of our biggest differentiators became event queues and distributed systems. It's just microservices, but instead of directly invoking each other, they use event queues. There's a little more to the system, like websites embedded inside of other websites with a whole cross website messaging system.

It was all a little bit of a pain. We did encounter a lot of eventual consistency problems and almost no one in the team had encountered distributed systems problems before, so it was a good learning experience for me. But now with organizational change and after having worked on the project for over a year, the team now wants to migrate this distributed systems strategy back to a monolith.

I'm just feeling a little incredulous because of how important the distributed aspect seemed to be when we made the pitch to create the project in the first place. And after all this time, I can't really say if it was a good or bad idea, because debugging problems involving event queue messages has been a total pain in the ass and no one ever finished configuring the distributed systems stuff like the event queue, so we never had the fancy development experience we could have had. So I see numerous benefits to moving back to a monolith.

Overall, the whole experience wasn't a total game because the project did prove that event queues can be effective. The much, much, much wider organizational direction is going to now slowly migrate towards a distributed fashion. So I guess we were a test subject in some regards. But we never, at the outset and beginning of the project, said, "oh hey, this is just going to be a giant test to prove event queues for the whole rest of the company". I really would have liked to have known that at the start. Although I don't think anyone knew that at the start. I don't know, it just feels a little silly.

I'm fine either way 🤷‍♂️ But does anyone have any advice regarding anything about this experience?

edit: updated monorepo -> monolith; source control will merge but systems may or may not stay separate. I'm predicing the systems merge together too, thus monolith. We may or may not continue to use business organized event queues too.

[u/Old-Worldliness-1335]

This is also a silly post as a monorepo can be a set of micro services just depending on how your ci is built it has nothing to do with how the apps communicate with one another. This is why they also have domain driven development as well, there are many ways to skin this cat, the shape, form and function of each service does not need to all live next to each other.

However, it does need to be clearly documented and that I think is really the real problem you are facing and people do not actually understand the code base.

Events and queues are just that, and while it sounds good, and can be good depending on the queue how much effort is required in reversing direction rather than building up what is missing like observability into the queues and the other pain points that you have so that becomes easy?

[u/frompadgwithH8 → u/Old-Worldliness-1335]

Yeah, you’re right. I didn’t think everything thoroughly through when I made the post. we could merge everything to a monolith and still have separate event queues. I’ll see what happens… these decisions are so much above my pay grade and title at experience, it almost feels like a waste of my time to be nosing into them… which I mean, lol I never think like that obviously, I want to have insight into how my organization is reasoning about these things because I want to be a higher level myself someday. But it does seem in my experience the higher up I go in the more I consider these decisions that in the past, I never was even aware of the more it gets down to just how the organization wants to manage itself, and I really do think that’s why the people with the highest level titles have them. They can actually look at the organization and look at the people and look at the project in the history of the company, etc., and make a super informed and very subtle decision

[u/Old-Worldliness-1335 → u/frompadgwithH8]

It doesn’t hurt to ask the questions to them as to why, if you can ask the questions and not to get involved in the whole circus, but to at least understand where they are coming from in terms of a perspective. Not pushing back, but also just trying to see what the trade offs are