Unexpectedly Intriguing!
20 January 2023

How can you train an AI?

By AI, we're referring to "artificial intelligence" systems, which are a special class of machine learning computer programs that are increasing showing up in some pretty amazing applications. Whether its generating an image based on text you enter or nearly instantaneously writing the equivalent of a school report on a particular subject, AI systems are leaving the world of science fiction and becoming today's reality.

But how do their developers train these systems to do these things?

Last year, Matt Parker visited Antartica, where he learned how to apply maths to identify specific humpback whales. The following 22-minute video describes how the mathematical methods developed for advanced image recognition made it possible for him to use an Excel spreadsheet to identify a specific whale he photographed swimming off the north coast of Antartica*.

Clearly, AI can deliver impressive results, but how far can you trust those results?

One area where photo-recognition AI systems could make a real impact is in radiology, where such systems could potentially diagnosis serious health conditions much more quickly at much lower cost than can be done by professional radiologists.

A recent study published in the British Medical Journal (BMJ) asked if AI could pass the Royal College of Radiologists' board examination. Spoiler alert: It couldn't, where why it couldn't tells us something about the limitations of these AI deep maching learning systems. Chuck Dinerstein of the American Council on Science and Health summarizes the study's main findings, in which the performance of AI-trained systems and human radiologists were compared (emphasis ours):

First, the obvious, with two exceptions, humans did better than the AI on diagnosis where both had been trained; when unfamiliar pathology was introduced, AI failed across the board. Second, while the humans fared better, theirs was not a stellar performance. On average, newly minted radiologists passed 4 of the ten examinations.

“The artificial intelligence candidate... outperformed three radiologists who passed only one mock examination (the artificial intelligence candidate passed two). Nevertheless, the artificial intelligence candidate would still need further training to achieve the same level of performance and skill of an average recently FRCR qualified radiologist, particularly in the identification of subtle musculoskeletal abnormalities.”

The abilities of an AI radiology program remain brittle, unable to extend outside their training set, and as evidenced by this testing, not ready for independent work. All of this speaks to a point Dr. Hinton made in a less hyperbolic moment.

“[AI in the future is] going to know a lot about what you’re probably going to want to do and how to do it, and it’s going to be very helpful. But it’s not going to replace you.”

Here's the kicker according to Dinerstein:

We would serve our purposes better by seeing AI diagnostics as a part of our workflow, a second set of eyes on the problem, or in this case, an image. Interestingly, in this study, the researchers asked the radiologists how they thought the AI program would do; they overestimated AI, expecting it to do better than humans in 3 examinations. That suggests a bit of bias, unconscious or not, to trust the AI over themselves. Hopefully, experience and identifying the weakness of AI radiology will hone that expectation.

Like any human expert, AI has limitations. Identifying and knowing what those limitations are will be key to determining how trustworthy they are. In the case of health care, as the example from radiology makes clear, it could be your health that's on the line if you blindly put more trust into a system than it deserves.

Reference

Shelmerdine, S.C.; Martin, H.; Shirodkar, K.; Shamshuddin, S.; Weir-McCall, J.R. "Can artificial intelligence pass the Fellowship of the Royal College of Radiologists examination? Multi-reader diagnostic accuracy study." BMJ 2022; 379. DOI: 10.1136/bmj-2022-072826. (Published 21 December 2022).

* If you know your geography, you already knew every coast of Antartica is the north coast!...

Labels: , , , ,

About Political Calculations

Welcome to the blogosphere's toolchest! Here, unlike other blogs dedicated to analyzing current events, we create easy-to-use, simple tools to do the math related to them so you can get in on the action too! If you would like to learn more about these tools, or if you would like to contribute ideas to develop for this blog, please e-mail us at:

ironman at politicalcalculations

Thanks in advance!

Recent Posts

Indices, Futures, and Bonds

Closing values for previous trading day.

Most Popular Posts
Quick Index

Site Data

This site is primarily powered by:

This page is powered by Blogger. Isn't yours?

CSS Validation

Valid CSS!

RSS Site Feed

AddThis Feed Button

JavaScript

The tools on this site are built using JavaScript. If you would like to learn more, one of the best free resources on the web is available at W3Schools.com.

Other Cool Resources

Blog Roll

Market Links

Useful Election Data
Charities We Support
Shopping Guides
Recommended Reading
Recently Shopped

Seeking Alpha Certified

Archives
Legal Disclaimer

Materials on this website are published by Political Calculations to provide visitors with free information and insights regarding the incentives created by the laws and policies described. However, this website is not designed for the purpose of providing legal, medical or financial advice to individuals. Visitors should not rely upon information on this website as a substitute for personal legal, medical or financial advice. While we make every effort to provide accurate website information, laws can change and inaccuracies happen despite our best efforts. If you have an individual problem, you should seek advice from a licensed professional in your state, i.e., by a competent authority with specialized knowledge who can apply it to the particular circumstances of your case.