r/ControlProblem • u/chillinewman approved • 2d ago

AI Capabilities News GPT-5 outperforms licensed human experts by 25-30% and achieves SOTA results on the US medical licensing exam and the MedQA benchmark

Post image
5 Upvotes

61% Upvoted

43 comments sorted by

8

u/Dmeechropher approved 2d ago

Does @deedydas mean to imply that the most useful, important, irreplaceable, and critical part of a doctor's job is passing a medical exam?

4

u/wren42 1d ago

Precisely. Better at taking a test doesn't mean it can replace the role. 

1

u/ZorbaTHut approved 1d ago

We let people act as doctors after they've passed medical exams. It's not literally the job, but we have historically used it as a sign they're ready for the actual job.

1

u/UpsetMud4688 1d ago

We have also used iq tests to measure intelligence, and eq tests for emotional intelligence. Llms being able to do well in these doesn't mean they are intelligent ,nor emotionally intelligent

1

u/ZorbaTHut approved 1d ago

Technically, but you'd need a pretty serious counterargument to claim that they aren't.

At some point the thing we're looking for with regards to "intelligence" is "can they produce the same answers that we would otherwise have to hire a smart person to produce".

1

u/UpsetMud4688 1d ago

can they produce the same answers that we would otherwise have to hire a smart person to produce".

That is not what intelligence is. That is a flawed shorthand we use to measure it, but the point of the tests are to latch on to an internal process, not just to complete the test itself

You can learn iq test answers and answer them in the same exact way a smart person not exposed to these tests before would. That does not mean you are just as intelligent, even though you are "producing the same answers we would otherwise need a smart person to produce"

0

u/ZorbaTHut approved 1d ago

That is not what intelligence is.

We don't know what intelligence is.

You can learn iq test answers and answer them in the same exact way a smart person not exposed to these tests before would.

You don't need a smart person to memorize stuff. You do need a smart person to answer new questions. And if you're producing the same answers that you would otherwise need a smart person to produce, i.e. not just memorizing answers, then that's close enough to intelligence for me.

1

u/UpsetMud4688 1d ago

you don't need to literally memorise the answers. You can practice for these tests and do the same as someone much smarter. This may be the same as being smarter to you, but it just isn't. Once again, these test are inherently flawed because they are shorthands, and dont directly measure intelligence

We don't know what intelligence is.

Therefore we can make up random bullshit and define it as intelligence

0

u/ZorbaTHut approved 1d ago

You can practice for these tests and do the same as someone much smarter.

Yes. What do you think most doctors do?

What they're saying here is "as smart as a doctor".

Therefore we can make up random bullshit and define it as intelligence

Better than taking actual accomplishments and defining it as not-intelligence.

1

u/ElectronicLab993 1d ago

By your logic anybody from the street who would pass this test could be a doctor I dont know about you but I wouldnt want to have a doctor like that

0

u/ZorbaTHut approved 1d ago

What extra relevant things do you think doctors need to do in order to become a doctor?

You know what they call a doctor who almost fails the medical licensing exam?

Doctor.

1

u/ElectronicLab993 1d ago

Practical test (licensing) and residency Would you allow for somebody to heal youw ho never diagnosed real person before? I swear to god some.of you AI fans have to be just young kids

1

u/ZorbaTHut approved 1d ago

Practical test (licensing)

Sure, so we'll have them do a practical test also. No biggie.

and residency

Well, get GPT to answer medical questions for a year or two, and it's done the equivalence of a residency.

Would you allow for somebody to heal youw ho never talked with real person before?

"Never talked with real person before"?

What specific skills do you think they'll be missing by merely being trained on millions of interactions with real people?

→ More replies (0)

1

u/U_Sound_Stupid_Stop 1d ago

It's not literally the job, but we have historically used it as a sign they're ready for the actual job.

Literally untrue.

No one becomes doctor just by passing exams, there are multiple stages of practical trainings notably the residency.

1

u/ZorbaTHut approved 1d ago

Which is extremely rare to fail, and if you insist, we can put AI through that as well.

1

u/U_Sound_Stupid_Stop 1d ago

The failure rate of humans is irrelevant since we're talking about entirely different beings, if they can even be called thus.

Yes, I would insist.

1

u/ZorbaTHut approved 1d ago

And if it passed the residency requirements, would you be fine with it?

1

u/U_Sound_Stupid_Stop 1d ago

Honestly, my only interest here is that your original statement was factually incorrect.

1

u/ZorbaTHut approved 1d ago

Which one, this one?

We let people act as doctors after they've passed medical exams. It's not literally the job, but we have historically used it as a sign they're ready for the actual job.

What's incorrect about it? The final step to become a doctor in the US is passing the USMLE.

1

u/Dmeechropher approved 1d ago

It's not the final step to practicing without oversight, which is what @deedydas is implying.

1

u/ZorbaTHut approved 1d ago

From what I understand, "pass the USMLE Stage Three" and "get someone to sign off on your residency being complete" are roughly parallel.

→ More replies (0)

1

u/Mad-myall 1d ago

The human brain is kinda fudgy on details so it's important a doctor succeeds on tests like this. An AI is just a fudgy database, so it can do tests like this easily simply because it's fed all the questions and answers of tests just like this. 

It might be able to enhance a doctors work by extending the information easily within reach, but cannot perform the actual task.

1

u/ZorbaTHut approved 1d ago

It might be able to enhance a doctors work by extending the information easily within reach, but cannot perform the actual task.

What, specifically, is it that a doctor can do that an AI can't do?

1

u/Mad-myall 1d ago

Well if it could do it now, we would already be seeing it, but to me it seems the AI's weaknesses are still hallucinations, and being unable to perform reasoning, or ascertain missing details.

It's hard to prove a negative 100% though, so how about instead you show evidence that an AI can do what a doctor does outside of written tests it was fed answers for!

1

u/ZorbaTHut approved 1d ago

Well if it could do it now, we would already be seeing it

Would we? If we had a reasonably simple statistical model that did better on average than doctors, how long do you think it would take for us to start using it systematically?

It's hard to prove a negative 100% though, so how about instead you show evidence that an AI can do what a doctor does outside of written tests it was fed answers for!

AI beats doctors at diagnosing illness, AI beats doctors at diagnosing rashes, AI beats doctors at diagnosing disease (these are three separate studies!) AI was beating radiologists back in 2018 and continued to do so in 2023. And 2 in 3 physicians using AI (as of half a year ago; the number is likely higher now).

What more evidence are you looking for?

1

u/Dmeechropher approved 1d ago

Doctors practice independently after an additional 2-4 years of residency, and candidate doctors are also required to do student rotations in hospitals satisfactorily before being allowed to take final exams.

Candidate doctors are also required to publish medical research.

So you're right, it is a sign that they're ready for the job. It's also the easiest to measure, fastest to finish, and least relevant to the work of the other half dozen historical signs a doctor is ready for the job.

If passing a medical exam was a good sign that someone would be an effective doctor, we wouldn't have a single bad doctor, eh? After all, they all passed the medical exam.

1

u/ZorbaTHut approved 1d ago

Doctors practice independently after an additional 2-4 years of residency, and candidate doctors are also required to do student rotations in hospitals satisfactorily before being allowed to take final exams.

Great, so let's get it started with a residency.

Candidate doctors are also required to publish medical research.

The doctors we care about are those who are expected to treat people, not do research. You're confusing medical doctors with PhD's.

(I acknowledge the terms are confusing.)

If passing a medical exam was a good sign that someone would be an effective doctor, we wouldn't have a single bad doctor, eh? After all, they all passed the medical exam.

Yup. Might be a pretty crummy doctor to start with. But it'll also be a far cheaper and more accessible doctor, and there's plenty of simple stuff that you don't need a highly skilled doctor for.

And if the objection is "it'll only be as good as human doctors", then that sounds like a great place to start.

1

u/pm_me_your_pay_slips approved 1d ago

I guess now we will have unqualified people passing the test with the help of AI, yay!

1

u/HatersTheRapper 1d ago

but can it make me a millionaire

1

u/IMightBeAHamster approved 1d ago

Benchmarks are benchmarks people. As anyone in any field will tell you, what works in theory often fails to work in practice.

1

u/Ok-Breakfast-3742 1d ago

Nice try gpt!