For space scientists evaluating massive datasets from ever-more-powerful space instruments, big data has grown to be a major difficulty. A team from the Southwest Research Institute has created a machine learning tool to effectively categorize huge, complicated datasets so that deep learning model can comb through them and find potentially dangerous solar occurrences in order to address this issue. The newly developed labeling method can be used or modified to handle further problems requiring huge datasets.
It is getting harder for scientists to interpret and assess pertinent patterns as space sensor packages collect complicated data in ever-growing amounts. Machine learning (ML), in which algorithms learn from past experience to generate judgments or predictions that can factor in more information simultaneously than humans can, is increasingly important for processing huge complicated datasets.
However, humans must first classify all the data, which is frequently a laborious task, in order to use ML approaches.
“Labeling data with meaningful annotations is a crucial step of supervised ML. However, labeling datasets is tedious and time consuming,” said Dr. Subhamoy Chatterjee, a postdoctoral researcher at SwRI specializing in solar astronomy and instrumentation and lead author of a paper about these findings published in the journal Nature Astronomy.
“New research shows how convolutional neural networks (CNNs), trained on crudely labeled astronomical videos, can be leveraged to improve the quality and breadth of data labeling and reduce the need for human intervention.”
We created an end-to-end, deep-learning approach for classifying videos of magnetic patch evolution without explicitly supplying segmented images, tracking algorithms, or other handcrafted features. This database will be critical in the development of new methodologies for forecasting the emergence of the complex regions conducive to space weather events, potentially increasing the lead time we have to prepare for space weather.
Dr. Derek Lamb
By extracting and learning complicated patterns, deep learning systems can automatically analyse and interpret vast amounts of complex data. The key precursors of space weather events are regions on the solar surface where strong, complex magnetic fields arise. The SwRI team analyzed videos of the solar magnetic field to locate these regions.
“We trained CNNs using crude labels, manually verifying only our disagreements with the machine,” said co-author Dr. Andrés Muñoz-Jaramillo, an SwRI solar physicist with expertise in machine learning.
“We then retrained the algorithm with the corrected data and repeated this process until we were all in agreement. While flux emergence labeling is typically done manually, this iterative interaction between the human and ML algorithm reduces manual verification by 50%.”
Iterative labeling approaches such as active learning can significantly save time, reducing the cost of making big data ML ready. Furthermore, by gradually masking the videos and looking for the moment where the ML algorithm changes its classification, SwRI scientists further leveraged the trained ML algorithm to provide an even richer and more useful database.
“We created an end-to-end, deep-learning approach for classifying videos of magnetic patch evolution without explicitly supplying segmented images, tracking algorithms or other handcrafted features,” said SwRI’s Dr. Derek Lamb, a co-author specializing in the evolution of magnetic fields on the surface of the Sun.
“This database will be critical in the development of new methodologies for forecasting the emergence of the complex regions conducive to space weather events, potentially increasing the lead time we have to prepare for space weather.”
