There are some very impressive AI/ML technologies that are already in use as part of existing medical software systems (think: a model that highlights suspicious areas on an MRI, or even suggests differential diagnoses). Further, other models have been built and demonstrated to perform extremely well on sample datasets.
Funnily enough, those systems aren’t using language models 🙄
(There is Google’s Med-PaLM, but I suspect it wasn’t very useful in practice, which is why we haven’t heard anything since the original announcement.)
Eh. Depends on which tech is being used and how. For a lot of things, relatively basic ML models purposefully trained do a pretty good job, and are, in fact, limited by the diagnoses in the training data. But more generalized “AI” tools seem rather… questionable.
Like, you can train a SVM on fMRIs to compare structures in the brain between patients diagnosed with bipolar disorder and those that are not diagnosed with it, and it will have an accuracy rate on new patients basically equal to the accuracy rate of the doctors who did the diagnosing in the training set. But you’ll have a much harder time creating a model that takes in fMRIs and reports back answers to the question of “which brain disease or abnormality do I have?”
This stuff works much closer to advertised when it’s narrowly defined and purpose built, but the people making and funding this work want catch-all doctor replacements, because of course they do, because there’s way more money in charging hospitals and patience 10% less than a doctor’s salary than there is in providing tools that make doctors’ efforts in diagnosing specific illnesses easier.
Precisely. Many of the narrowly scoped solutions work really well, too (for what they’re advertised for).
As of today though, they’re nowhere near reliable enough to replace doctors, and any breakthrough on that front is very unlikely to be a language model IMO.
That and bias, absolutely need improvements. That doesn’t mean LLMs can’t be extremely effective if given appropriate tasks. The problem is that the people who make decisions about where they’re used aren’t technical enough to understand their strengths and limitations
I don’t think technical knowledge gives as good a sense as a lot of experience working with one.
Like saying the guys who designed a particular car would know best how it’ll perform on various racetracks. My sense is a driver would have a better sense.
Peak intelligence, is realizing an LLM doesn’t care whether its tokens represent chunks of text, sound, images, videos, 3D models, paths, hand movements, floor planning, emojis, etc.
The keyword is: “multimodal”.
As for being able to correctly correlate some “chunks of MRI scan” with the word “tumor”… that’s all about the training (which I’d bet Claude is missing… did I hear “investment opportunity”? Guy isn’t wrong).
I had to prepare a high level report to a senior manager last week regarding a project my team was working on.
We had to make 5 professional recommendations off of data we reported.
We gave the 5 recommendations with lots of evidence and references to why we came to that decision.
The top question we got was: “What are ChatGPT’s recommendations?”
Back to the drawing board this week because LLMs are more credible than teams of professionals with years of experience and bachelor-masters level education on the subject matter.
“It came up with more or less the same recommendations. Though it didn’t fully understand the specific target goals of your project, so our recommendations are more complete and actionable ready.”
I mean, as long as you are the one prompting ChatGPT, you can probably get it to spit out the right recommendations. Works until they fire you because they are convinced AI made you obsolete.
You joke but several sections of our HR department got cut and replaced with Enterprise GPT-4. We talk to an internal chatbot now about HR questions and some forms.
It did the opposite lmao. I asked it what my vacation leave was because you need to verify leave amounts before you’re allowed to request any additional leave. It said I had 0 in my balance and I know for a fact I have at least a week left 🤪 took almost a month to sort it out. Had to provide balance screenshots and everything. I’d be probably fucked if I hadn’t manually screenshot my leave amounts beforehand.
It is quite terrifying that people think these unoriginal and inaccurate regurgitators of internet knowledge, with no concept of or heuristic for correctness… are somehow an authority on anything.
All you need to succeed on this planet is the self confidence to say things. It literally does not matter the accuracy. It’s how you express it. I wish I knew this when I was younger. I’d cut out all the imposter syndrome that held me back.
I wish it was that easy. If you go too long it’s boring, and if you’re too confident you sound arrogant. At this point I’ve kind of just accepted there are people who can sell, and that I’m not one of those people.
I think this depends on the crowd. Unfortunately, the intelligent crowd and the crowd with money and power is not exactly the same. Though hopefully there is overlap.
Using a Large Language Model for image detection is peak human intelligence.
A picture is worth a thousand words
Well, image models are getting better at producing text, just sayin’
I read the same thing in Nevvsweeek.
AI cars are still running over pedestrians and people think computers are to the point of medical diagnosis?
They are already used in medicine reliably. Often. Welcome to the future. Computers are pretty good tools for many things actually.
There are some very impressive AI/ML technologies that are already in use as part of existing medical software systems (think: a model that highlights suspicious areas on an MRI, or even suggests differential diagnoses). Further, other models have been built and demonstrated to perform extremely well on sample datasets.
Funnily enough, those systems aren’t using language models 🙄
(There is Google’s Med-PaLM, but I suspect it wasn’t very useful in practice, which is why we haven’t heard anything since the original announcement.)
I have read some headline that said that some of these models just measure age of a patient and a quality of the machine making photos.
Eh. Depends on which tech is being used and how. For a lot of things, relatively basic ML models purposefully trained do a pretty good job, and are, in fact, limited by the diagnoses in the training data. But more generalized “AI” tools seem rather… questionable.
Like, you can train a SVM on fMRIs to compare structures in the brain between patients diagnosed with bipolar disorder and those that are not diagnosed with it, and it will have an accuracy rate on new patients basically equal to the accuracy rate of the doctors who did the diagnosing in the training set. But you’ll have a much harder time creating a model that takes in fMRIs and reports back answers to the question of “which brain disease or abnormality do I have?”
This stuff works much closer to advertised when it’s narrowly defined and purpose built, but the people making and funding this work want catch-all doctor replacements, because of course they do, because there’s way more money in charging hospitals and patience 10% less than a doctor’s salary than there is in providing tools that make doctors’ efforts in diagnosing specific illnesses easier.
Or, at least there is if you can pull it off.
Precisely. Many of the narrowly scoped solutions work really well, too (for what they’re advertised for).
As of today though, they’re nowhere near reliable enough to replace doctors, and any breakthrough on that front is very unlikely to be a language model IMO.
And they should no more replace doctors in the future than x-ray machines did in the past. We should never want them to.
Really.
Says all you need to know about their opinion lol
Still AI misalignment is a real issue. I just don’t remember which model was studied and had been found out that it was missaligned.
That and bias, absolutely need improvements. That doesn’t mean LLMs can’t be extremely effective if given appropriate tasks. The problem is that the people who make decisions about where they’re used aren’t technical enough to understand their strengths and limitations
I don’t think technical knowledge gives as good a sense as a lot of experience working with one.
Like saying the guys who designed a particular car would know best how it’ll perform on various racetracks. My sense is a driver would have a better sense.
Peak intelligence, is realizing an LLM doesn’t care whether its tokens represent chunks of text, sound, images, videos, 3D models, paths, hand movements, floor planning, emojis, etc.
The keyword is: “multimodal”.
As for being able to correctly correlate some “chunks of MRI scan” with the word “tumor”… that’s all about the training (which I’d bet Claude is missing… did I hear “investment opportunity”? Guy isn’t wrong).
I had to prepare a high level report to a senior manager last week regarding a project my team was working on.
We had to make 5 professional recommendations off of data we reported.
We gave the 5 recommendations with lots of evidence and references to why we came to that decision.
The top question we got was: “What are ChatGPT’s recommendations?”
Back to the drawing board this week because LLMs are more credible than teams of professionals with years of experience and bachelor-masters level education on the subject matter.
“It came up with more or less the same recommendations. Though it didn’t fully understand the specific target goals of your project, so our recommendations are more complete and actionable ready.”
“How much did your recommendations cost to produce?”
I mean, as long as you are the one prompting ChatGPT, you can probably get it to spit out the right recommendations. Works until they fire you because they are convinced AI made you obsolete.
you fool
“these are chatgpt’s recommendations we just provided research to back them up and verify the ai’s work”
Haha and then the conversation would then be “Yes but can we see ChatGPT’s research?”
That’s when you drop trou, bend over, spread the cheeks, and ask them to let you know when they’re done reviewing ChatGPT’s “research”.
My butt is much too perky for these goons. They don’t deserve it.
“What do we pay you guys for then? You are all fired and Tummy the intern will do everything with ChatGPT from here on out!”
You joke but several sections of our HR department got cut and replaced with Enterprise GPT-4. We talk to an internal chatbot now about HR questions and some forms.
You should see if you can get it to hallucinate a pay raise or 3 months vacation.
It did the opposite lmao. I asked it what my vacation leave was because you need to verify leave amounts before you’re allowed to request any additional leave. It said I had 0 in my balance and I know for a fact I have at least a week left 🤪 took almost a month to sort it out. Had to provide balance screenshots and everything. I’d be probably fucked if I hadn’t manually screenshot my leave amounts beforehand.
Jesus christ that thing’s real??
Why can’t they just use a simple calendar app system where you book it off??? Who would use a large language model for that rubbish?
You work for a crazy company, my friend.
That is the least worst implementation!
I knew one HR person who cared about employees and did her best to help out. She only lasted 6 months.
It is quite terrifying that people think these unoriginal and inaccurate regurgitators of internet knowledge, with no concept of or heuristic for correctness… are somehow an authority on anything.
Only thing you need to do to realise how bad they are is to play Chess against it. Vs using a chessbot from 30 years ago, it really shows.
That’s a meaningless comparison.
All you need to succeed on this planet is the self confidence to say things. It literally does not matter the accuracy. It’s how you express it. I wish I knew this when I was younger. I’d cut out all the imposter syndrome that held me back.
You, and we, are better off for it.
The issue is that it’s been forgot (Remember the 5th of November)
I wish it was that easy. If you go too long it’s boring, and if you’re too confident you sound arrogant. At this point I’ve kind of just accepted there are people who can sell, and that I’m not one of those people.
I think this depends on the crowd. Unfortunately, the intelligent crowd and the crowd with money and power is not exactly the same. Though hopefully there is overlap.
I think this points to a large problem in our society is how we train and pick our managers. Oh wait we don’t. They pick us.