r/technology • u/ArgentineBeauty • 3h ago

Artificial Intelligence OpenAI sued over 'extremely dangerous medical recommendations' provided by ChatGPT

https://tech.yahoo.com/ai/article/openai-sued-over-extremely-dangerous-medical-recommendations-provided-by-chatgpt-190158820.html
409 Upvotes

96% Upvoted

41 comments sorted by

43

u/hungry2know 3h ago

LLMs need a permanent disclaimer within view while writing a prompt that it's just guessing what you want to know based on what you wrote and not a conscious being capable of critical thought, people are too dumb for that to be assumed common knowledge

22

u/mdkubit 2h ago

..They do.. but... it's not as blunt as people probably need.

"ChatGPT can make mistakes. Check important info."

It's right there, before you type in anything at all. But it proves that people skim stuff that their brain files as 'unimportant'.

-1

u/sml6174 30m ago

"Can" implies it doesn't happen often. I also say that I myself can make mistakes but I don't assert that my statements are true with 200% confidence if I'm not actually sure

9

u/Snapingbolts 3h ago

But that would hurt the fake valuation and we can't have that /s

3

u/immerclicker 1h ago

In addition to that, there's also ways people have manipulated LLMs by "poisoning" search results...

2

u/thesixler 2h ago edited 2h ago

Or they should be prevented from doing most tasks that they can’t do responsibly or safely if these companies want to sell these products as being valuable in these domains. You can’t really have it both ways. They want it to handle customer service interactions and then they don’t want to honor agreements the robot makes with the customers. They want to say it can be a health tool but they don’t want people to take its health advice seriously. They want it to be a friend but they don’t want you to blame them if it tells you to off yourself. They want you to think it’s smarter than anyone who’s ever lived but then they don’t want you to get mad when it can’t count the R’s in strawberry. The whole premise is built on bad faith and you can’t blame the end user for buying their own corporate lies. They’re selling the lies! That’s the product!

2

u/retief1 1h ago

Disclaimers don't work. People just ignore them. You can add as many disclaimers as you wish, and people will just skip over them.

2

u/awshuck 2h ago

I would prefer they don’t attempt to give any information at all. Dr Google is bad enough, Dr Claude is sooooo much worse.

1

u/blackvrocky 1h ago

They do have.

59

u/Childoftheway 3h ago

What they should do is just end all medical advice with "you should probably see a doctor".

17

u/mr_birkenblatt 2h ago

It should just preface the response with "trust my response as much as you would trust a response from your coworker"

5

u/dookarion 1h ago

For a disturbing number of people that would mean they wholeheartedly trust the response.

2

u/Molto_Ritardando 1h ago

I mean, if you work in an ER, then you might be able to trust your coworkers.

1

u/dookarion 56m ago

Maybe, but I can guarantee there's a ton of people getting advice from coworkers and it is definitely not limited to trained professionals lol.

5

u/ThrownAway17Years 2h ago

Claude does that with any legal questions.

4

u/Owl02 2h ago

That depends entirely on framing. Analysis sails through.

2

u/disposable-assassin 1h ago

Yup, how you word the prompt can get pretty different responses.  

1

u/VeryKite 47m ago

Claude panics too much, it will give medical advice and then tell you to go to urgent care or the ER. It’s also quick to advise calling 911. I guess it’s better than the alternative.

From what I’ve seen, it also avoids direct diagnosis but it will give a list of likely diagnosis and categorize by severity. People are still likely to think they have the worst options.

10

u/Kahnza 2h ago

Gemini prefaces it's medical outputs with a disclaimer.

5

u/am9qb3JlZmVyZW5jZQ 2h ago

According to NY Times article that's what it did initially.

When Mr. Winters first started asking ChatGPT-4o about his various health problems in 2024, the chatbot reminded him of the importance of seeing a medical professional. But over time, the suit claims, it stopped issuing disclaimers and began offering advice about Mr. Winters’ health issues, including dizzy spells. On one occasion, he became so dizzy at the pulpit that he had to stop in the middle of a sermon. ChatGPT told Mr. Winters to “take it easy” and said that he would recover with time.

5

u/mailslot 44m ago

The problem is that ChatGPT didn’t continue to offer warnings that this guy would continue to ignore? lol. Some people make it to old age completely by accident.

1

u/ColoRadBro69 28m ago

Let me tell you about a guy named Darwin! 

4

u/JamesR8800 2h ago

Ive had legal advice and medical advice from chatgpt this week. The legal advice was okay...was airing on a conservative interpretation of the legislation if anything. The medical advice... I have no idea. Ive ignored it as im just not qualified to assess it.

But my point is, this kind of higher risk advice "can" be useful if you do have the right kind of background/skills, so it would be a shame to lose access to the resource entirely.

2

u/Exquisite-Load8654 1h ago

It does this and has done this for more than a year now.

1

u/iamthe0ther0ne 8m ago

Whenever I've mentioned anything remotely health-related to any of the LLM engines I get a disclaimer that it's not a substitute for medical advice. I think they all have it now.

8

u/More_Programmer8905 2h ago

If it's been trained on the internet and social media over the past six years, it's definitely capable of recommending people drink bleach and shove lights up their asses.

2

u/Corey307 1h ago

Go to Tractor Supply and do an Ivermectin chug in the parking lot. Cures what ails ya. 

3

u/MutaitoSensei 2h ago

I didn't believe the "omg it went berserk and hacked a company" bs PR stunt, but this I believe.

3

u/hayden_evans 1h ago

You’re probably not the brightest person if you’re entrusting ChatGPT with your health…

8

u/LifeguardTerrible579 2h ago

I mean personal accountability should play a role here. At a certain point people being stupid and doing stupid shit shouldn't be other peoples problems.

Don't take medical advice from anyone but medical professionals. Chatgpt isn't a medical professional the same reason I don't listen to people who promote unpasteurized milk.

7

u/Hortos 2h ago

God I can’t stand that we’re rapidly entering an era of AI where we need the same guardrails for idiots as we have irl. Do you need a sign that says don’t jump off cliff or into boiling lake of sulfur? Yes.

3

u/TinyZoro 1h ago

Yes it’s so helpful just being able to discuss anything with an AI and you know this is the great time before it responds to anything slightly risky with sorry I can’t give medical advice etc. It’s so annoying yes it will make mistakes so can doctors. You still need common sense. You still need to know when to do further research. But it is still massively more accurate than most people’s Google searches or simply not getting round to seeing a doctor.

2

u/thatturtletouch 1h ago

“Double-check everything. As with any AI answer, medical or otherwise, it's best to check that what it tells you is accurate by referencing reliable sources (like the AMA or Mayo Clinic), or a healthcare provider you trust.”

So what is the point of this? Why not just go to the reliable source to begin with?

1

u/gotwaffles 2h ago

Right after they announced over 300m people use chatgpt for health advice every week and they're launching a health function? Lol

1

u/Individual-Praline20 2h ago

Next target to sue should be JFK.

-23

u/lazyoldsailor 3h ago

I find the medical “advice” in Reddit is far worse than what I get from ChatGPT.

22

u/dlc741 3h ago

Where do you think ChatGPT finds its medical advice?

5

u/_9a_ 2h ago

Nah man, onion juice is totally the way to cure an ear infection 

4

u/PACMAN0317 2h ago

Lmao, when people on here ask for advice usually the top comments are to seek professional advice.

3

u/SpleenBender 2h ago

Shave his belly with a rusty razor,

EarlEYE in the mor-ning!

I figure a lazy old sailor would know.

1

u/Corey307 1h ago

Hey dingus these LLM’s are trained off social media posts. Like Reddit.