Unique collaboration between electrical engineers and ophthalmologists at UC San Diego solves clinical and computational challenges

[JRC]

For the past five years, several University of California San Diego electrical engineering graduate students have spent one day a week at Joan and Irwin Jacobs Retina Center at Shiley Eye Institute. Jacobs Retina Center is the only freestanding retina research center in the country, working to increase the understanding of retinal diseases, such as macular degeneration and diabetes-related blindness, and to conduct clinical trials to develop better treatments for these diseases. But the students aren’t there for treatment, and they’re not there to see patients.

Through a unique and long-standing collaboration, the electrical engineers are embedded in Jacobs Retina Center to partner with ophthalmologists to develop better computer vision, artificial intelligence (AI), and image processing tools to help physicians diagnose patients faster and more accurately; predict which drugs will be most successful for specific patients; and even aid in the process of developing new therapeutic treatments for retinal diseases.

“We extend our gratitude to Joan and Irwin Jacobs for their funding, which underscores the importance of supporting centers of excellence, and to the NIH for recognizing the value of this work," said Chancellor Pradeep K. Khosla. "Such cross-school collaborations enhance both education and research. Our world-class Jacobs School of Engineering and Ophthalmology department, along with our exceptional faculty, have facilitated this highly productive work, promising improved diagnosis and treatments for retinal diseases.”

Over the course of five years, the researchers have collaborated on 21 papers, publishing advances in both clinical and engineering journals.

“We started out with the question of whether we could help ophthalmologists align images of the retina, and whether AI can actually help doctors be faster and more accurate with their detection of disease,” said Truong Nguyen, professor of electrical and computer engineering at the Jacobs School of Engineering at UC San Diego. “And we have done that. In this collaboration it has been very critical to have both the expertise on the engineering side, with machine learning, image processing and so on, and the expertise from the clinical side. We have been very successful in terms of getting results, and have made a big, broad impact.”

William Freeman, MD, Distinguished Professor and vice chair of the Viterbi Family Department of Ophthalmology at UC San Diego School of Medicine and director of Jacobs Retina Center at Shiley Eye Institute, partnered with graduate students in Nguyen’s lab and international ophthalmology fellows at the retina center. Freeman and Nguyen add that such a longstanding and impactful partnership, benefitting both fields of research, is extremely unique.

“It’s exceptionally rare to have weekly discussions where engineers actively engage with patient care, making it easier to understand their work and needs,” said Freeman. “Our collaborations aren’t just one-offs; even after someone may work with us for two to three years and earns their PhD, the ongoing connection is unique. This sustained effort, bridging medicine and engineering across campus, isn’t just about claiming that we use AI – it’s a dedicated approach to tackling practical health care challenges through innovation.”

[OCTA]

OCTA images from eye vasculature impacted by age-related macular degeneration (left) and a healthy vasculature from a patient whose AMD is in remission.
OCTA images from eye vasculature impacted by age-related macular degeneration (AMD) at left, and a healthy vasculature from a patient whose AMD is in remission on the right. The researchers developed an AI tool that allows ophthalmologists to glean the same information from photos that they would previously have needed to perform a biopsy to assess. Figures from a paper published in the journal Retina.

Developing AI tools for physicians to diagnose retinal disease more accurately

While this collaboration has yielded nearly two dozen scientific publications to date with more in the pipeline, there are several particularly noteworthy accomplishments. Most recently, the joint team developed an AI tool that was able to predict whether a patient had age-related macular degeneration just by looking at Optical Coherence Tomography (OCT) angiography images of a patient’s blood vessels in their eye. These images are noninvasive and taken in several minutes during a standard clinical visit. The AI tool allows ophthalmologists to glean the same information from photos that they would previously have needed to perform a biopsy to assess. And it outperformed human experts with 80% percent accuracy based on images alone.

Not only was the tool able to help doctors diagnose patients faster and more accurately, but it could also be used to develop drugs better able to treat macular degeneration.

“This tool was able to tell us that actually these vessels are causing the disease,” said Dr. Anna Heinke, a retina specialist from Poland and Germany who is completing a post-doctoral retina fellowship. She is first author of the paper, published in the journal Retina. “This would be an interesting endpoint for clinical trials if you wanted to design drugs targeting these vessels. Even as physicians, we couldn't tell if these are active or inactive cases of macular degeneration by looking at the images, but there is something the AI is learning that allows it to do that. It will also be interesting for explainable AI to actually know what it is that the AI is seeing that determines the disease state.”

In another paper, the team devised a way to synthesize multiple images from different time points to more accurately check if there has been growth in the size of blood vessel damage or a tumor, for example.

“When you're looking at retina diseases in the retinal periphery, or the outer edges, they are difficult to see,” said Freeman. “For example, it may be challenging to determine if a retinal tumor is growing because you're viewing a small picture in the corner. To track changes over time, typically you would compare images from the previous and current years to see if it has grown past a specific point, like a blood vessel or marker.”

The engineers developed a method to overlap previous images with the current image of a patient’s eye so that it’s immediately evident whether the object in question has grown or changed from previous years. The AI method localized the spot in question 37 percent faster than the traditional side-by-side image comparison, and with a 0 percent error rate, compared to 18 percent error rate in a side-by-side comparison by ophthalmologists. Dirk-Uwe Bartsch, Adjunct Professor of Ophthalmology and Co-Director, Jacobs Retina Center; Cheolhong An, Assistant Adjunct Professor in the Department of Electrical and Computer Engineering; and Nathan Scott, MD, assistant professor of ophthalmology at UC San Diego School of Medicine and ophthalmologist at UC San Diego Health, are also involved in this research effort. 

The ophthalmologists and engineers have developed deep learning networks to correct eye motion in 3D retinal imaging; quantitatively evaluate morphological changes in vasculature due to age related macular degeneration using OCTA angiography; corrected distortions between ultra-widefield and narrow-angle retinal images; and created an AI tool to overlay multimodal images from different optical instruments in patients with retinitis pigmentosa, among many other joint advances. 

[cavichini]

Previously, ophthalmologists would compare images of the retina from different time points side-by-side, as seen in the top checkerboard images, to detect changes in the vessels or growth in tumors or disease damage. Thanks to the collaboration with electrical engineers, the team devised an AI tool to synthesize images from multiple time points into one image, and spot changes in the image over time to more rapdily and accurately measure changes.

Engineering for the public good

Bo Wen, an electrical engineering graduate student who is collaborating on these ophthalmology projects, said having such tangible results has been a huge motivator for him and the other electrical engineering students involved.

“If we’re not actually helping people, then why are we doing this?” Wen said. “If we are only here to get some papers published, that’s too superficial. We want to have our work make an impact in disease diagnosis and treatment.”

Nguyen encouraged other faculty and students at UC San Diego to invest the time to develop collaborations across campus. With nearly 4,000 faculty members working on leading-edge research, there are endless opportunities to apply your expertise in a new way.

“The real power of this collaboration is that we understand our expertise, we understand our contributions, and we regularly meet with one another to see how we can both leverage that expertise,” said Nguyen. “There are many opportunities for engineers to branch out and really serve and make an impact on real problems - that’s what I love the most about this.”

OCTA images from eye vasculature impacted by age-related macular degeneration (AMD) at left, and a healthy vasculature from a patient whose AMD is in remission on the right. Figures from a paper published in the journal Retina.
Previously, ophthalmologists would compare images of the retina from different time points side-by-side, as seen in the top checkerboard images, to detect changes in the vessels or growth in tumors or disease damage. Thanks to the collaboration with electrical engineers, the team devised an AI tool to synthesize images from multiple time points into one image, and spot changes in the image over time to more rapdily and accurately measure changes.

[fundus]

In yet another joint paper, the engineers and ophthalmologists
In yet another joint paper, the engineers and ophthalmologists used AI to overlay images from different optical instruments in patients with retinitis pigmentosa. The ability to align the retinal multimodal images helps ophthalmologists better understand the relationship between measures of retinal function and structure in these patients.
Charting new territories of privacy, responsibility with AI
To train the artificial intelligence and deep learning algorithms which their tools are based on, the engineers need thousands, and sometimes tens of thousands, of images of the retina in different states of disease and health to teach the model what to look for and how to detect certain diseases and progressions.

The team can create these tools in part because they have greater access to labeled, reliable image data than nearly any other research groups in the field. The international ophthalmology fellows who spend between one to three years working at Jacobs Retina Center before returning to their home institutions often retain close ties, and their home institutions send anonymized images to the team at Jacobs Retina Center. Nearly a dozen eye care centers around the world collaborate on different projects and send images to be used in training the models for specific projects.

While the images are all anonymized and carry no patient information, there are still privacy concerns for what could unfold in the future. The team would like to eventually release the tools they’ve developed for use by the broader ophthalmology community but are waiting to see how the field ultimately balances issues of privacy with data availability.

For example, current guidance is that the iris– what is scanned at the airport if you have Clear, for example– is a uniquely identifiable portion of the eye, but fundus images of the retina are not identifiable and therefore not protected health information. However, it’s unclear if that will hold true in the future, and what it would mean if such a dataset and tool had already been released to the broader public.

“You can identify someone by their retinal vessels if you know what they look like,” said Freeman. “Even though it’s not common practice yet, with infrared cameras available, it’s now easy to take infrared scans of people. There’s potential for these scans to become more widespread. We need to carefully consider the possibilities before making these valuable tools more accessible.”

To date, the FDA has approved just one clinical ophthalmology AI platform, for a diabetic retinopathy screening tool. However, the algorithm was trained and works only on images from one specific camera and one specific device. UC San Diego is developing tools that could accept images from multiple different cameras, but whether such a tool could earn FDA approval is unclear.

The clinicians also are adamant that these algorithms and models are ultimately just tools, and are meant to enhance, not replace, the expertise and ethical decision making of ophthalmologists.

“We look at this as a tool to help physicians, but not replace their decision,” said Heinke. “Ultimately, the physician is responsible for the patient, not AI. I’m still going to look at the output of the AI, and say ok I agree with that, the screening shows its diabetic retinopathy; or, here’s where I need more information. At the end, the physician must look at the output and accept or reject it.”

Overall, Freeman and Nguyen note that cross-school collaborations show the strengths and depth of scientific expertise at UC San Diego, and the university’s unique ability to work across different scientific fields to solve pressing challenges.

Learn more about research and education at UC San Diego in: Artificial Intelligence