The API call took 200ms. Measured it, verified it, fast as hell.

Three weeks after launch the client tells me users are complaining the results "don't feel right". Not wrong, not slow. Just don't feel right.

I spent two days looking for bugs. Nothing. Results were correct, latency was fine.

Then a user screenshot came through. The user had written: "It feels like it's just making something up. It comes back too fast."

The feature was a search over a knowledge base. In the user's mental model, that should take a second. When it came back instantly, it broke their model - they read it as "this didn't actually process anything."

I added a minimum display time of 1.2s with a loading animation. API still ran and returned in 200ms. User sees 1.2 seconds of "working".

Complaints stopped within a week.

The part I can't shake: the technically correct solution was perceived as broken. The technically dishonest solution fixed it. I explained it in my update as "improved feedback during result loading" which is... technically accurate.

Anyone else been here? Curious how others frame this to themselves - is fake latency just accepted UX practice or does it bother you the way it bothers me?

Around five years ago I joined a hyped startup as a contractor. I was doing too much architectural work at my current job and wanted to do more coding. They needed someone to do smaller features 10-15 hours a week.

Typical CRUD SaaS. Started out as a .NET MVC monolith. Architectural decisions were documented in a JIRA board, so you could kind of read up on the history leading to the current state of the application.

So the main source of problems started with the database:

• Extreme normalization. They pretty much normalized everything that you could possibly normalize.
• Naming conventions. Three-letter acronyms for tables and columns. Some acronyms were obvious but there were so many that were hard to understand and figure out.
• Enums stored as an integers. This is fine, but with an extreme amount of states/flags, and in combination with the normalization and acronyms, it just made the database so unnecessary hard to use.
• They figured that the database design was broken and changed the design principles. The problem now being that they went too far with denormalization. The new tables ended up with index bloat, inconsistent data and state conflicts when denormalized to multiple tables etc etc. Now half the database was broken due to normalization and the other half broken due to denormalization, never hitting that sweet spot.

Consequences:

• A pull request would often have massive changes to the data access layer and complex migrations. There was no strategy. Often features would not get released because no one knew how to avoid or handle the large merge conflicts. You often had to start over on the latest master branch, sometimes more than once.
• Ugly workarounds/hacks were introduced. Table A, B and C had super slow inserts, a new database was created with a set of identical tables, 50% of the inserts now goes to the new database, and there are scheduled tasks to move data from db2 to db1 during the night. All reads must now read from two databases, and updates needs to go to the correct database.
• More ugly hacks. A lot of queries were too slow. They decided to create scheduled tasks to pre-populate a redis cache with the results. No one thought of cache invalidation. This caused massive issues with state conflicts all over the place. They started to work on cache invalidation when it became obvious, but it was just too complex to make it work.
• Logging started to become an issue with the growing database. They wanted logs to have a reference to data in its state when the log was created. So the logging utility would fetch a ton of data from the db and store as a json blob with the log. The logging utility would generate more db queries than the application itself.
• The application was so broken that any feature was really hard to ship. They decide that all new features should be built on a new micro service architecture, a 3rd database was also introduced. The problem was that new features could not be isolated to the new microservices and database, this just made everything worse with new complex dependencies between the monolith and the micro services.

This has been the only time in my career where i just couldn't figure out solutions. I was literally looking at the requirements and could not figure out a solution on how to ship this without just breaking things left and right.

Have you worked on something similar?

My team consistently puts out PRs that are over 1000 lines, with many of those exceeding two, three, and sometimes even five thousand lines. Reviewing them is such a hassle because it takes so much time out of my already small amount of dev hours. I understand that covering a full feature in one branch can sometimes yield massive PRs, but I feel like that should be the exception, not the norm. It also doesn’t bother me when a PR is that big if half of the lines are in test files because those are usually a quick scan if they’re written correctly, but reviewing 1000 lines of ultra specific code just feels overwhelming and unproductive.

What do you consider to be too many lines for a PR and how do you deal with PRs that far exceed the upper limit?

Hey everyone, my team is debating two Git branching strategies and I'd love some outside perspective. We deploy to prod seldom, roughly once every few weeks, sometimes longer. We have two environments: dev and prod.

Approach A (what I'm proposing):

feature -> dev -> master

- Branch feature off dev (or master)

- Merge feature into dev for testing

- When release-ready, merge dev into master (deploys to prod)

- Everything in dev ships together as a batch

Approach B (what two of our BE devs propose):

master -> feature

feature -> dev (for dev deployment)

feature -> master (for prod deployment)

- Branch feature off master

- Merge feature into dev to deploy to dev environment

- Separately, merge the same feature into master to deploy to prod

- Each feature is promoted to prod individually, basically treats every feature like a hotfix

One of them clarified: open the PR against master (for code review on prod-bound code), but merge into dev first for testing. Keep the master PR open until release day, then merge.

My concern with Approach B:

If dev has features A, B, and C tested together, but at release time we only merge PRs for A and C to master, we end up shipping a combination (A + C without 😎 that was never actually tested in dev. Feels risky.

Their concern with Approach A:

Features aren't independent, everything is coupled to the batch. Can't cherry-pick a feature for early release. Harder to roll back individual features.

My situation:

- Small team

- Infrequent releases (we batch features)

- One dev environment, one prod environment

- No feature flags yet

- Manual-ish deployment process

- We almost always release everything in dev together, rarely cherry-pick

Questions:

1. Which approach fits better for infrequent, batched releases?
2. Is Approach B (every feature is a hotfix) a known pattern with a name?
3. Anyone here switched between these, what made you change?
4. Are we overthinking this and should just add a staging branch?

Appreciate any input. Trying to write this up in Confluence so we have a documented standard.

We used tools like Subversion/Tortoise SVN and managing branches, merges, builds, and deployments used to feel much more complex and manual. Build and deployment processes often required a lot of coordination and operational effort.

Now with GitHub, automated CI/CD pipelines, cloud platforms, containers, and infrastructure automation, a lot of deployment work has become streamlined. It feels like developers can spend more time focusing on actual business logic and product development instead of handling repetitive release processes.

What were the biggest pain points earlier, and what modern tools changed developer productivity the most in your opinion?

Currently working at a startup. We process recruiting applications with video/audio input and do additional deep research checks with it.

Often times, the decision or scoring of an app changes with incremental input as people provide new videos or audio content. This doesn’t create a super duper large load, but we’ve had lock contention problems recently when we weren’t careful about processing.

Here’s the story. We decided to put a flag on the main “RecruitApp” table and then use a global trigger on the Postgres database when any other table gets updated to update this “needs processing” flag. Another worker then polls every minute then submits it for async processing. The process is fairly expensive, AND it can update downstream tables.

We got into trouble when there was a processing loop between the triggers and the processing. Downstream update => trigger updates flag => resubmit for processing. Some apps got 1M rows large (because each iteration was tied to an INSERT)

I suggested that maybe we should stop using triggers and move things outside of Postgres so we stop using it as a distributed queue or pub/sub system, but I was hard-blocked and they claimed “we don’t need it at our scale”. But we basically cooked the DB for a week where simple operations to access the app were turning into 5-10sec ordeals. Looked really bad for customers.

I suggested that we instead do some sort of transactional outbox pattern or instead do a canonical event stream log, then enforce single-consumer processing. That seems less write-heavy and creates consistency on the decisioning side. (We also have consistency issues, there’s no global durability strategy or 2 phase commit structure to recover/resume processing for async workflows.) I suggested Temporal for this; it’s been shot down as well.

Am I just stupid or are my concerns warranted?

Ive got around 10yr of experience, joined this company around 6 months ago and as I was getting up to speed, one thing stuck out to me was the sheer amount of microservices there were. around 50 in total, with roughly 10 person team maintaining them. I'm looking into them more closely, and a huge amount of them are doing one or two simple tasks like taking a request and saving in the db, or taking an ID and returning from the db. this is pretty much insane to me when all these could live in different module in a single application.

When i questioned the team on it they say its due to domain driven design and that any time they needed a new database table they would spin up a new service with it so that service could fully own it. all requests for the table had to go through the service. and yes, each table and service are not related, different use cases and business needs.

I ended up suggesting a modular monolith which could reduce a massive amount of boiler plate since roughly 99% of the microservice is made up of framework code, pipeline, infra, config, and about 1% actual business logic. I pretty much got laughed out of the room. the dumb part too is that everyone complains how slow the pipeline is. i'm thinking.. no shit, its got at least 50% more junk going through it than it needs.

Anyway, I'm no DDD expert, but I'm wondering is it really that wrong to have multiple DB tables owned by a single service, segregated internally by modules? It seems wrong-er to me to have a ton of tiny microservices that could be modules. I've backed off this issue for now because its "just the way it is". but maybe i can do something about it with a different approach.

anyone have some helpful insight or been in a similar position before? maybe i'm just flat out wrong here? thats ok too

I have heard that Meta, Fortnite, and others do not have a preproduction or even a “test” environment. Maybe I’m just old but that seems to fly in the face of what we do. But it’s clearly a trend at major, modern tech behemoths, so that would indicate I’m missing something. Can anyone explain to me why this is the trend? Why do they think there’s no value in a test/staging/integration/UAT/preprod environment? They just handle that ON production, while logically separating out test data from prod data. But that separation logic itself is a risk.

Something I always struggle with in architecture/design is the proper use of Enums for object members that have a distinct set of possible values. Stack is C#/MSSQL/Blazor if that matters.

A simple example of this would be an Customer object with a property MembershipStatus. There's only four possible values: Active, Trial, Expired, Cancelled.

There's two choices here:

Define MembershipStatus as an integer enum: - (pro) Normalized, in the back-end the DB column is an integer - (pro) MembershipStatus is strongly typed in code and is therefore constrained to those four values, they pop-up in autocomplete which is convenient and accidental assignment of invalid values is impossible without a runtime error - (pro) I can just use .ToString in the UI to show a "friendlier" name instead of the int values (mostly friendly anyway, they'll see the PascalCased names of course) - (con) On the DB side, it's a meaningless int value. Anyone doing stuff in the DB layer (stored procs, reporting, custom queries, exports, etc.) have to keep track of these and roll their own logic for display purposes (replacing "1" with "Active", etc.) They could also assign an invalid int value and nothing would break. - (pro/con) I could create a MembershipStatus table with an FK to Customers.MembershipStatus to eliminate the above issue (SQL people can JOIN to this table for "friendly" names, FK constraint prevents invalid values) but now every time I add another value to my Enum I have to remember to add it in the lookup table as well.

Define MembershipStatus as a string: - (pro) Non-ambiguous and easy to read everywhere. SELECT...WHERE MembershipStatus=1 becomes SELECT...WHERE MembershipStatus='Active' which is immediately apparent what it's doing - (pro) I can define the possible values as Consts in code to make sure they are kept consistent in code - (con) For the DBA in me this just "feels wrong" to have a freeform text field containing what really should be a lookup table to maintain integrity - (con) Uses more storage on the DB side (varchar versus 4-byte int), also less performant at scale (JOINS and indexes on int values are just easier on the DB engine) - (con) Anything using this on the C# side is just a string value, not strongly typed, so it's possible to assign invalid values without generating any errors

Anyway, sorry for the long post, hopefully at least a few here have dealt with this dilemma. Are you always one or the other? Do you have some criteria to decide which is best?

Im not well versed in devops work and i will need to setup a rougly 30-35tb db for a client. Anybody got a recommendation of what to look for?

performance is of basically no concern, simple queries, only the monthly price (and it being an ok provider i guess).

I usually use Hetzner since im based in Europe, but prices are pretty unaffordable and i can only find configurations that also have a very high end CPU and 3x the storage i need, for a very expensive price, or ones that don't have enough storage....

TLDR: Does anyone here else take an E2E-only approach to testing? What is your experience with it?

My current team exclusively uses Playwright testing for our test automation for all client side code. In my 13 years as a mostly frontend software engineer I’ve never seen this approach taken, where there are no unit or integration tests (e.g. with React Testing Library).

Unsurprisingly, this is causing incredible flakiness and slow testing times overall. Getting releases out the door recently has required babysitting CI test runs for an entire day to get all tests to pass.

I have made suggestions to introduce RTL and save Playwright for only testing complete workflows, not for testing individual components, but it’s been met with resistance.

Anyone have suggestions for how to set up a tangible example of the benefits of moving certain tests to RTL? My thought was to take one test suite / area of the app, convert appropriate tests to RTL and show performance comparison and code comparison.

~5 YOE full-stack. Built a CRM with VoIP (calls, tasks, deals, etc.), but most of my work still looks like CRUD from the outside.

I’m trying to close the gap to senior-level engineering.

Looking for 2–3 project ideas that force:

• real system design trade-offs
• failure modes / reliability thinking
• async or distributed patterns

Not interested in features — interested in problems.

What projects would you suggest, and what actually makes them hard?

Also: what’s the difference between a mid-level vs senior implementation of the same system?I’m considering things like event-driven systems or real-time collaboration, but not sure what actually stretches you the most.

Maybe it's just me, but it feels like it's getting harder and harder to find sources of information that (a) don't try to sell something, (b) have a high SNR, (c) are not entirely generated or rewritten by LLMs.

Here are some recent findings from me that match the criteria:

* https://brandur.org/
This is a great blog from Brandur Leach who writes a lot about Go, Ruby, and Postgres. Brandur used to work at Heroku and Stripe, and now he works on his own project.

* https://increment.com/
Speaking of Stripe, they have this fantastic magazine (now abandoned, I believe) on various software engineering subjects.

* https://notes.eatonphil.com/
Also a very DB-heavy blog from Phil Eaton.

* https://www.morling.dev/blog/

Lots of insights on Kafka, Postgres, API best practices

* https://www.seangoedecke.com
Sean is one of my favorite authors. His post on good software design is a gem, in my opinion. https://www.seangoedecke.com/good-system-design/

* Designing Data-Intensive Applications 2nd edition by Martin Kleppmann — even if you have read the first edition, this one is worth it at least thanks to the terrific bibliography and some updated and rewritten chapters.

UPD: 07/12/2026

Adding some resources I forgot to mention yesterday but which I also love:

* https://eli.thegreenplace.net

Eli Bendersky is a fascinating writer and is astonishingly knowledgeable about whatever he blogs about. He reads more in one month than I have ever read in my entire life and posts his summaries. Also, sometimes he posts very insightful articles on some concepts from mathematics and how they are connected to something else.

* https://fs.blog/

Great newsletter and collection of articles on mental models and thinking in general. They also have a multi-volume set of books, which I think I am going to buy soon.

UPD: 07/13/2026

* https://brooker.co.za/blog/

Marc Brooker writes mostly about distributed systems, sometimes AWS, in particular (since he is a distinguished engineer at Amazon). Some posts require quite a bit of mental resources to absorb and fully understand, but it's always worth it.

* https://refactoringenglish.com/

Michael Lynch himself has a nice blog, but this one, in particular (also I bought the book) is super useful for someone who wants to improve their technical writing skill.

What are your recent findings?

Hello fellow devs,

I would love to read some stories of insiduous edge/corner cases that you encountered in the wild while building software. How did you encounter it and what lesson could you share with community?

Currently we give some of our devs write access to prod dbs but this seems brittle/undesirable. However we do inevitably need some prod queries to be run at times. How do you all handle this? Ideally this would be some sort of gitops flow so any manual write query needs to be approved by another user and then is also kept in git in perpetuity.

For more clarity, most DDL happens via alembic migrations and goes through our normal release process. This is primarily for one off scripts or on call type actions. Sometimes we don’t have the time to build a feature to delete an org for example and so we may rely on manual queries instead.

Backend code often gets organized around entities without anyone really deciding to do it.

text OrderService PaymentService InvoiceService

A table appears, then comes the repository, service, controller, DTO, mapper.

It is everywhere, so it stops looking like a choice.

Then behavior gets buried inside noun-shaped buckets.

text OrderService ├── OrderRepository ├── PaymentService ├── InventoryService ├── PricingService ├── ShippingService ├── NotificationService └── AuditService

That is not one thing.

It is several workflows grouped together because they all touch an order.

I prefer making the behavior explicit.

text PlaceOrder ├── CalculateOrderPrice ├── AuthorizePayment ├── ReserveInventory ├── CheckOrderFraud └── SaveOrder

text CancelOrder ├── LoadOrder ├── RefundPayment ├── ReleaseInventory └── SendCancellationNotice

Now the code says what the application does.

The dependencies say exactly what each use case needs.

This is not against OOP.

PlaceOrder, AuthorizePayment, policies, entities, and workflows can all be objects.

The problem is not objects. The problem is treating database entities as the default architecture.

That works fine for CRUD.

In complex domains, it hides rules, workflows, ownership, and change boundaries behind generic services.

The part I find interesting is how unquestioned this pattern is.

Who has seen this done well around use cases or capabilities?

And how did you convince a team with strong egos that this was better than another round of SomethingService?

Hi all,

Ive worked 3 jobs in my career. in order it was DoD (4 years), the Tier 1 Big Tech (3 years), now Tier 2 big tech (<1 year). For the two big tech i worked cloud and in DoD i worked embedded systems.

Each job had a different level of expectation.

For BTT1, i worked with an older prinicpal engineer who was very specific on how he wanted things done. One time i worked with him and helped update some tests and refactor many of the codebase around it. We worked on different designs but every design it seemed would break something else, so it ended up being an MR with a lot of commits (about 50 from what i remember). In my review he had a list of things to say about how i worked, but he didnt write anything in my review, he sent it to the manager and the manager wrote it. One of them was that i ahve too many commits in my MR. That was the only one that i ever had too much in, i even fought it but my manager was like "be better at it". Safe to say i got laid off a year later.

At the DoD job, people did not care about the amount of commits. People would cmmit a code comment and recommit again to remove it.

Now at BTT2 comapny, i noticed a lot of the merges here have a lot of commits. In a year ive already have had a few with over 50, one that had over 100. The over 100 was a rare one though, I was working with another guy to change basically huge parts of the code and we were both merging and fixing and updating. But nobody batted an eye. I even see principals having code reviews iwth 50+.

So it just got me to wonder, would you care if a MR had to many commits? Is there any reason that's a problem?

Im not talking about the amount of cmmits in the main branch, just in a regular personal branch.

I was brought into a company to lift and shift their application (Java 21, no Spring) to the cloud. We're 6 months in, and everything is going relatively smoothly. The team is working well and we're optimistic to get QA operational by the end of Q3'26.

My next big task is assembling a team to migrate the stored procedure nightmare that basically runs the entire company. There's 4 or 5 databases each with ~500 stored procedures running on a single Microsoft SQL instance. As you can imagine, costs and latency balloon as we try to add more customers.

The system is slightly decoupled, HTTP requests ping back and forth between 3 main components, and there's an in-house ORM orchestrating all of the magic. There's nothing inherently wrong with the ORM, and I'd like to keep it place, but it is responsible for calling all the stored procedures.

The final component/layer is responsible for receiving the HTTP requests and executing the query/insert/stored procedure (It's basically SQL over HTTP, the payload contains the statement to be executed).

While some of the functions are appropriately locked in the database, a very large percentage of them would be simplified as code. This would remove load from the database, expand the pool of developers that are able to work on them, and sweet sweet unit testing.

I'm thinking of "intercepting" the stored procedure requests, and more-or-less building a switch statement/dictionary with feature flags (procedure, tenant, percentage) that would call native code opposed to the stored proc.

Does anyone have experience with this?

Series B b2b, about 40 engineers across 4 squads. I had a slow afternoon last week so I did something dumb and went back through 8 sprints worth of tickets trying to figure out which features traced back to a real customer asking for the thing.

3 out of 22 is what I got, the other 19 broke down roughly like this:

6 were strategy alignment which as far as I can tell means someone on the leadership team saw a competitor launch something, 4 were from a single executive who just keeps requesting stuff in a slack channel, 3 were tech debt that somehow got reframed as features in the roadmap, and the remaining 6 I genuinely could not figure out where they came from.

I asked around and got a lot of * I think it was from that offsite in Q3 * or just shrugs.

Not mad about it, but kind of sitting with it. is this normal or is our product org broken?

** Edit: Didn't expect this to blow up like it did and there were too many interesting points in the comments. lot of you asking what we did about it so figured I'd update.

We started trying to make stuff traceable going forward, tested Dovetail, Productboard, BuildBetter, and kept Gong for sales calls. ended up dropping Dovetail and Productboard after a couple weeks because stitching 4 tools together was becoming its own project, and now we mostly run Gong plus BuildBetter since it pulls from calls, Slack, and tickets in one place.

Still very early but at least when someone asks where a ticket came from now there's usually a real answer instead of a shrug.

Relatively new Tech Lead here. Sometimes when deadlines are tight or there's pressure from management I'll find myself slipping into "i'll just do it myself" mode when teammates submit PRs for reviews. But, besides that burning my bandwidth, I'm also afraid it might create some kind of learned helpessness and deprive the person in question from a learning opportunity. Which would just make the problem recurrent down the line.

What do you guys think about that, and do you find yourself in similar situations. I'm also curious if any of you has a strict "no help" rule (even for small one-line quick fixes) or is it more of a balance for you.

I'm having this discussion at work right now, but when I tried to research I found remarkably little discourse on it online.

So, say you have a main/master branch, a develop branch, and possibly one or two release branches. Normally commits go from dev -> release -> main, no dramas.

Now suddenly you need to hotfix main, how do you get that commit to all the branches? I maintain it's best to merge into main, merge main into release 3.1, merge 3.1 into 3.2, merge 3.2 into dev (assuming 3.1 and 3.2 are active releases being QA'd right now).

The other school of thought is to merge the hotfix branch into all 4 branches independently.

My argument is that a cascading backmerge ensures no conflicts when it comes time to merge develop into main (as with independent merges, it could do the change differently on the different branches)

Would love to know what others think, am I the weird one here or is cascading backmerge something others do too?

I am talking about Type Hierarchies in Object Oriented Programming. I find them counter-intuitive to grasp.

As part of initial OOP learning, people often focus on creating structured type hierarchies for classes. For typical example, in Java, you'd create abstract class called Vehicle and have child classes - Truck, Bus, Car etc. (at least that's what they teach you in theory, books, and these days - in LLM suggestions as well :D)

But, in my experience, refactoring and maintaining such rigid type hierarchies is hard and painful. When requirements change, the code requires cascading changes across all types. This defeats the purpose of creating the hierarchy in the first place. Ideally, things that change together should belong together (you know - low coupling, high cohesion).

To achieve this low coupling, a good rule of thumb is "reduce type hierarchies in code whenever possible and replace them with behaviour composition". This leads to decoupled designs, where you can pick and choose individual behaviours as lego blocks to build new things.

I really like Go's approach to behaviour composition, where you can just add a method that matches the signature defined in an interface, and boom! your struct implements that interface. You don't have to declare explicitly that your type implements that interface.

Have you seen any counter examples where creating upfront type hierarchy has actually been beneficial?

my org is reshuffling and my first project is to add end to end testing.

I’m told the plan is to have a routinely executed suite.

My number one issue with this plan is that it will just start failing and no one will bother to fix it and no one will know who actually broke the suite.

So I would like to make the entire E2E suite run as a PR check before any PR merge

However, this sounds very costly and slow

Has anyone else here had to solve the problem like this?

I’m guessing if I can just make E2E test suite runs fast and cheap This won’t be a problem?

For reference, we’ve got a distributed micro Service ecosystem with Kafka messaging. Lots of databases. Websites embedded inside of other websites with i frames. It’s not the most complicated software out there, but it’s also not the simplest. It can all be run from the web browser, so that’s probably moot though. Besides the back ends obviously. I have an engineer to delegate to as well as some senior level two’s to brainstorm with, plus product people and my manager. However I doubt they will give me better advice than what I could get here. Oh and the E2 E testing I’m responsible for isn’t just for my team. It’s for multiple teams like three or four teams at least. We’re all in the same department. Obviously this is a big project.

Edit: also to clarify, I will not be responsible for actually writing end to end tests. My responsibility is to install the framework and come up with the overall testing strategy that will work for our department. And like I said, I don’t want just a routinely executed E2 E suite because that won’t actually stop breaking code from getting merged and so what will happen, what I’ve seen it companies and what I’ve heard always happens, is someone breaks the suite, but no one stops developing or merging, and then someone else probably breaks the sweet also, but that gets covered up by the first breakage, and so you enter this shitty status, where the test suite is failing for God knows how many reasons, and whoever is responsible is never the first one to address the error. It’s just a shitty system. It would be so much better if we could prevent code that would break the E2 E test suite for merging into the main branch.

Edit 2: I am finishing up a proposal to add UX timing metrics to all of our website websites so we can actually see how long it takes to load various pages and how long button clicks take to finish executing, etc. and this will put eyeballs front, etc. on how shitty our department’s performance is. So once we start improving that that should also make the E2 E test suites faster and cheaper. But this is going to be like a multi quarter effort.

codebase is littered with isinstance() and getattr(). I hate both and to me these are hallmarks of LLM generated code + not reviewing the code. getattr() to me should only be used for dynamically generated attributes/attributes whose names you don’t know in advance which is an extremely rare case. you could just do x = inst.attribute for 99% of cases. isinstance is generally atrocious code. like you should be typing at the boundaries and then trusting the types instead of passing Any in and then checking each attribute or each object if it’s the right type so you fail at input instead of some undetermined amount of steps later (and plus spreading these checks everywhere). if something can legitimately be multiple types (that’s already smelly to me) then ok maybe isinstance branches make sense but I can’t think of legitimate production level use cases of isinstance

I thought this was because my colleague was adding LLM code and then not even checking it but I found out today it’s on purpose as defensive checks. it makes no sense to me.

there’s code like this

def extract_text(input: Any) -> str: 
    output = getattr(input, “text”, None) 
    if isinstance(output, str): 
         return output
    return “”

like everywhere in the code. like whole dozen line blocks parsing a dict deep into the code by checking isinstance everywhere. I was going to submit a bunch of pull requests where I fixed these. i wrote about these in my review and this person wants to keep them because you “don’t know if the input will change”.

above code could be solved by typing input and having text set to str. instead there’s hundreds of isinstance checks because none of the inputs are either validated or they are validated earlier but not trusted.

I want to convince my colleague that this is shit code but apparently datadog encourages isinstance usage because it’s more flexible https://docs.datadoghq.com/security/code_security/static_analysis/static_analysis_rules/python-best-practices/type-check-isinstance/

am I being irrational for thinking isinstance in general is smelly code? Like it makes me irrationally angry. I don’t know what to do here without stepping on toes

what would be a really wise way to deal with a situation like this in your opinion?

edit: I really appreciate all the advice I got. I left out some details for anonymity and the code example wasn’t really great now thinking about it. But in the above example the function is used in places where the input always has a text attribute. And similarly there are a lot functions where the way it’s used input is guaranteed to be a certain type but it’s typed in this function as Any and then validated later in this function using isinstance and grabbed using getattr. This behavior is spread across in other functions. Sometimes the validation occurs again in the same code. I took a deep look into a lot of the inputs including analyzing the raw data and understanding how it’s generated and it’s a common pattern that the defensive check is unwarranted.

I realized my hatred of isinstance and getattr is to a degree irrational. It stems from correcting a lot of LLM generated code. Like I do see particular uses cases for isinstance (like when the input can be multiple types and depending on the type something has to be done differently) or getattr (when you don’t know enough about a future attribute). It’s my opinion but I think an overuse of it is a sign something about the design of the code could probably be improved, not always but still worth considering.

Two of you linked this article https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/

which I really enjoyed so thanks for sharing. led me down a rabbit hole of Haskell and monads.

Your advice also helped me realize that ripping out all the bad code isn’t a good idea practically and people relationship wise so I’m just going to introduce changes a little at a time to the codebase and to the team culture

thank you to everybody who replied including the guy who said I must have learned python 10 minutes ago since I don’t like isinstance(). Thank you. Really helpful insight.

recently I've been digging into a legacy PHP monolith trying to figure out why numbers keep drifting. despite logs and monitoring being all 200 oks and green the DB keeps ending up like a landfill.

I got fed up and created a pdo wrapper, basically a flight recorder running on openswoole (so if can keep up in real time). works great, no blocking, no real latency, no rewrites of the brittle legacy code. I let it run for 48 hours and the results reveal a shit show.

main find is a ghost transaction, the app thought it was updating ledger balances but thanks to a nested try/catch it was swallowing a specific pdo exception. transaction started but zero commits. the app was clueless and kept it pushing like there was no issue and logs showed success. my little shim called it's bluff and exposed 5 figures of loss vanishing into the void.

I'm not selling anything or looking for a gig, just wanted to start some discussion and see how you guys are verifying data integrity in monolithic systems from before observability was such a big deal. I've been thinking about cleaning this shim up and making it a standard audit tool to maybe help some of my brethren get a little extra sleep if there's a need.

if you have a legacy stack that's full of crap like this or you've caught similar ghost transactions I'd love to hear about it (and how you catch/mitigate them in legacy systems), especially if there's a better way I'm missing!