How can you train an AI?
By AI, we're referring to "artificial intelligence" systems, which are a special class of machine learning computer programs that are increasing showing up in some pretty amazing applications. Whether its generating an image based on text you enter or nearly instantaneously writing the equivalent of a school report on a particular subject, AI systems are leaving the world of science fiction and becoming today's reality.
But how do their developers train these systems to do these things?
Last year, Matt Parker visited Antartica, where he learned how to apply maths to identify specific humpback whales. The following 22-minute video describes how the mathematical methods developed for advanced image recognition made it possible for him to use an Excel spreadsheet to identify a specific whale he photographed swimming off the north coast of Antartica*.
Clearly, AI can deliver impressive results, but how far can you trust those results?
One area where photo-recognition AI systems could make a real impact is in radiology, where such systems could potentially diagnosis serious health conditions much more quickly at much lower cost than can be done by professional radiologists.
A recent study published in the British Medical Journal (BMJ) asked if AI could pass the Royal College of Radiologists' board examination. Spoiler alert: It couldn't, where why it couldn't tells us something about the limitations of these AI deep maching learning systems. Chuck Dinerstein of the American Council on Science and Health summarizes the study's main findings, in which the performance of AI-trained systems and human radiologists were compared (emphasis ours):
First, the obvious, with two exceptions, humans did better than the AI on diagnosis where both had been trained; when unfamiliar pathology was introduced, AI failed across the board. Second, while the humans fared better, theirs was not a stellar performance. On average, newly minted radiologists passed 4 of the ten examinations.
“The artificial intelligence candidate... outperformed three radiologists who passed only one mock examination (the artificial intelligence candidate passed two). Nevertheless, the artificial intelligence candidate would still need further training to achieve the same level of performance and skill of an average recently FRCR qualified radiologist, particularly in the identification of subtle musculoskeletal abnormalities.”
The abilities of an AI radiology program remain brittle, unable to extend outside their training set, and as evidenced by this testing, not ready for independent work. All of this speaks to a point Dr. Hinton made in a less hyperbolic moment.
“[AI in the future is] going to know a lot about what you’re probably going to want to do and how to do it, and it’s going to be very helpful. But it’s not going to replace you.”
Here's the kicker according to Dinerstein:
We would serve our purposes better by seeing AI diagnostics as a part of our workflow, a second set of eyes on the problem, or in this case, an image. Interestingly, in this study, the researchers asked the radiologists how they thought the AI program would do; they overestimated AI, expecting it to do better than humans in 3 examinations. That suggests a bit of bias, unconscious or not, to trust the AI over themselves. Hopefully, experience and identifying the weakness of AI radiology will hone that expectation.
Like any human expert, AI has limitations. Identifying and knowing what those limitations are will be key to determining how trustworthy they are. In the case of health care, as the example from radiology makes clear, it could be your health that's on the line if you blindly put more trust into a system than it deserves.
Reference
Shelmerdine, S.C.; Martin, H.; Shirodkar, K.; Shamshuddin, S.; Weir-McCall, J.R. "Can artificial intelligence pass the Fellowship of the Royal College of Radiologists examination? Multi-reader diagnostic accuracy study." BMJ 2022; 379. DOI: 10.1136/bmj-2022-072826. (Published 21 December 2022).
* If you know your geography, you already knew every coast of Antartica is the north coast!...
Labels: ethics, health care, ideas, math, technology
Welcome to the blogosphere's toolchest! Here, unlike other blogs dedicated to analyzing current events, we create easy-to-use, simple tools to do the math related to them so you can get in on the action too! If you would like to learn more about these tools, or if you would like to contribute ideas to develop for this blog, please e-mail us at:
ironman at politicalcalculations
Thanks in advance!
Closing values for previous trading day.
This site is primarily powered by:
CSS Validation
RSS Site Feed
JavaScript
The tools on this site are built using JavaScript. If you would like to learn more, one of the best free resources on the web is available at W3Schools.com.
Other Cool Resources
Blog Roll
