The potential of artificial intelligence (AI) and machine learning is being used by MBARI and other research organizations in a new collaborative effort to speed up ocean research. Researchers urgently need to learn more about the organisms, ecosystems, and processes that make up the ocean in order to control the effects of climate change and other challenges.
The collection of images, videos, and other visual data vastly outpaces researchers’ ability to analyze it, which poses a fundamental challenge for scientists and engineers as they create sophisticated robotics that can visualize marine life and environments to track changes in the health of the ocean.
FathomNet is an open-source image database that uses state-of-the-art data processing algorithms to help process the backlog of visual data. The analysis of underwater imagery will no longer experience a bottleneck thanks to artificial intelligence and machine learning, which will also hasten crucial ocean health studies.
“A big ocean needs big data. Researchers are collecting large quantities of visual data to observe life in the ocean. How can we possibly process all this information without automation? Machine learning provides a pathway forwards, however these approaches rely on massive datasets for training. FathomNet has been built to fill this gap,” said MBARI Principal Engineer Kakani Katija.
Project co-founders Katija, Katy Croff Bell (Ocean Discovery League), and Ben Woodward (CVision AI), along with members of the extended FathomNet team, detailed the development of this new image database in a recent research publication in Scientific Reports.
The use of artificial intelligence in ocean research has been constrained by the absence of a standard set of existing images that could be used to train the machines to recognize and catalog underwater objects and life. Recent advances in machine learning enable fast, sophisticated analysis of visual data. FathomNet addresses this need by aggregating images from multiple sources to create a publicly available expertly curated underwater image training database.
While FathomNet is a web-based platform built on an API where people can download labeled data to train novel algorithms, we also want it to serve as a community where ocean explorers and enthusiasts from all backgrounds can contribute their knowledge and expertise and help solve challenges related to ocean visual data that are impossible without widespread engagement.
Kakani Katija
“In the past five years, machine learning has revolutionized the landscape of automated visual analysis, driven largely by massive collections of labeled data. ImageNet and Microsoft COCO are benchmark datasets for terrestrial applications that machine-learning and computer-vision researchers flock to, but we haven’t even begun to scratch the surface of machine-learning capabilities for underwater visual analysis,” said Ben Woodward, co-founder, and CEO of CVision AI and a co-founder of FathomNet. “With FathomNet, we aim to provide a rich, interesting benchmark to engage the machine-learning community in a new domain.”
Over the past 35 years, MBARI has recorded nearly 28,000 hours of deep-sea video and collected more than 1 million deep-sea images. This trove of visual data has been annotated in detail by research technicians in MBARI’s Video Lab.
MBARI’s video archive includes approximately 8.2 million annotations that record observations of animals, habitats, and objects. This rich dataset is an invaluable resource for researchers at the institute and collaborators around the world.
FathomNet incorporates a subset of MBARI’s dataset, as well as assets from National Geographic and NOAA.
The National Geographic Society’s Exploration Technology Lab has been deploying versions of its autonomous benthic lander platform, the Deep Sea Camera System, since 2010, collecting more than 1,000 hours of video data from locations in all ocean basins and in a variety of marine habitats. These videos have subsequently been ingested into CVision AI’s cloud-based collaborative analysis platform and annotated by subject-matter specialists at University of Hawaii and OceansTurn.
National Oceanic and Atmospheric Administration (NOAA) Ocean Exploration began collecting video data with a dual remotely operated vehicle system aboard NOAA Ship Okeanos Explorer in 2010.
More than 271 terabytes are archived and publicly accessible from the NOAA National Centers for Environmental Information (NCEI). NOAA Ocean Exploration originally crowd-sourced annotations through volunteer participating scientists, and began supporting expert taxonomists in 2015 to more thoroughly annotate collected video.
“FathomNet is a great example of how collaboration and community science can foster breakthroughs in how we learn about the ocean. With data from MBARI and the other collaborators as the backbone, we hope FathomNet can help accelerate ocean research at a time when understanding the ocean is more important than ever,” said Lonny Lundsten, a senior research technician in MBARI’s Video Lab, co-author, and FathomNet team member.
As an open-source web-based resource, other institutions can contribute to and use FathomNet instead of traditional, resource-intensive efforts to process and analyze visual data. MBARI launched a pilot program to use FathomNet-trained machine-learning models to annotate video captured by remotely operated underwater vehicles (ROVs). Using AI algorithms reduced human effort by 81 percent and increased the labeling rate tenfold.
Machine-learning models trained with FathomNet data also have the potential for revolutionizing ocean exploration and monitoring. For example, outfitting robotic vehicles with cameras and improved machine-learning algorithms can eventually enable automated search and tracking of marine animals and other underwater objects.
“Four years ago, we envisioned using machine learning to analyze thousands of hours of ocean video, but at the time, it wasn’t possible primarily due to a lack of annotated images. FathomNet will now make that vision a reality, unlocking discoveries and enabling tools that explorers, scientists, and the public can use to accelerate the pace of ocean discovery,” said Katy Croff Bell, founder and president of the Ocean Discovery League and a FathomNet co-founder.
As of September 2022, FathomNet contained 84,454 images, representing 175,875 localizations from 81 separate collections for 2,243 concepts, with additional contributions ongoing. FathomNet aims to obtain 1,000 independent observations for more than 200,000 animal species in diverse poses and imaging conditions eventually more than 200 million total observations.
For FathomNet to reach its intended goals, significant community engagement including high-quality contributions across a wide range of groups and individuals and broad utilization of the database will be needed.
“While FathomNet is a web-based platform built on an API where people can download labeled data to train novel algorithms, we also want it to serve as a community where ocean explorers and enthusiasts from all backgrounds can contribute their knowledge and expertise and help solve challenges related to ocean visual data that are impossible without widespread engagement,” said Katija.
