The amount of time, effort, and money needed to train ever-more-complex neural network models is soaring as researchers push the limits of machine learning. Analog deep learning, a new branch of artificial intelligence, promises quicker processing with less energy use.
Similar to how transistors are the essential components of digital computers, programmable resistors are the fundamental building blocks of analog deep learning. Researchers have developed a network of analog artificial “neurons” and “synapses” that can do calculations similarly to a digital neural network by repeatedly repeating arrays of programmable resistors in intricate layers. Then, using difficult AI tasks like image recognition and natural language processing, this network may be trained.
The goal of a diverse MIT research team was to increase the speed of a particular kind of artificial analog synapse that they had previously created. They used a useful inorganic substance in the manufacturing process to give their devices a speed boost of a million times over earlier iterations, which is roughly a million times faster than the synapses in the human brain.
This inorganic component also contributes to the resistor’s exceptional energy efficiency. The new material is compatible with silicon production methods, in contrast to materials employed in the earlier iteration of their device. This modification has made it possible to fabricate nanometer-scale devices and may open the door to their incorporation into commercial computing hardware for deep-learning applications.
“With that key insight, and the very powerful nanofabrication techniques we have at MIT.nano, we have been able to put these pieces together and demonstrate that these devices are intrinsically very fast and operate with reasonable voltages,” says senior author Jesús A. del Alamo, the Donner Professor in MIT’s Department of Electrical Engineering and Computer Science (EECS). “This work has really put these devices at a point where they now look really promising for future applications.”
“The working mechanism of the device is the electrochemical insertion of the smallest ion, the proton, into an insulating oxide to modulate its electronic conductivity. Because we are working with very thin devices, we could accelerate the motion of this ion by using a strong electric field, and push these ionic devices to the nanosecond operation regime,” explains senior author Bilge Yildiz, the Breene M. Kerr Professor in the departments of Nuclear Science and Engineering and Materials Science and Engineering.
“The action potential in biological cells rises and falls with a timescale of milliseconds, since the voltage difference of about 0.1 volt is constrained by the stability of water,” says senior author Ju Li, the Battelle Energy Alliance Professor of Nuclear Science and Engineering and professor of materials science and engineering, “Here we apply up to 10 volts across a special solid glass film of nanoscale thickness that conducts protons, without permanently damaging it. And the stronger the field, the faster the ionic devices.”
The working mechanism of the device is the electrochemical insertion of the smallest ion, the proton, into an insulating oxide to modulate its electronic conductivity. Because we are working with very thin devices, we could accelerate the motion of this ion by using a strong electric field, and push these ionic devices to the nanosecond operation regime.
Bilge Yildiz
These programmable resistors greatly accelerate neural network training while also significantly lowering the cost and energy required. This might speed up the process through which researchers create deep learning models that can be used for things like fraud detection, self-driving cars, or picture analysis in medicine.
“Once you have an analog processor, you will no longer be training networks everyone else is working on. You will be training networks with unprecedented complexities that no one else can afford to, and therefore vastly outperform them all. In other words, this is not a faster car, this is a spacecraft,” adds lead author and MIT postdoc Murat Onen.
Co-authors include Frances M. Ross, the Ellen Swallow Richards Professor in the Department of Materials Science and Engineering; postdocs Nicolas Emond and Baoming Wang; and Difei Zhang, an EECS graduate student. The research is published today in Science.
Accelerating deep learning
Analog deep learning is faster and more energy-efficient than its digital counterpart for two main reasons. “First, computation is performed in memory, so enormous loads of data are not transferred back and forth from memory to a processor.”
Parallel processes are also carried out by analog processors. An analog processor doesn’t require more time to perform new operations as the size of the matrix increases because all computation happens simultaneously.
The key element of MIT’s new analog processor technology is known as a protonic programmable resistor. These resistors, which are measured in nanometers (one nanometer is one billionth of a meter), are arranged in an array, like a chess board.
Learning occurs in the human brain as a result of the strengthening and weakening of synapses, the connections between neurons. This approach, where the network weights are programmed by training algorithms, has been used by deep neural networks for a long time. Analog machine learning is possible with this new processor by varying the electrical conductivity of protonic resistors.
The conductance is controlled by the movement of protons. More protons are pushed into a resistor channel to increase conductance, and more protons are pulled out to decrease conductance. This is done by conducting protons but blocking electrons in an electrolyte (similar to a battery’s electrolyte).
The researchers investigated various materials for the electrolyte in order to create a protonic resistor that is programmable, extremely quick, and highly energy efficient. While other devices used organic compounds, Onen focused on inorganic phosphosilicate glass (PSG).
PSG, the powdery desiccant agent used to eliminate moisture and found in little bags in new furniture boxes, is essentially silicon dioxide. It is also the most popular oxide utilized in the production of silicon. To create PSG, silicon is given unique properties for proton conduction by adding a small amount of phosphorus.
Onen hypothesized that an optimized PSG could have a high proton conductivity at room temperature without the need for water, which would make it an ideal solid electrolyte for this application. He was right.
Surprising speed
Because PSG has a large number of nanometer-sized pores with surfaces that act as pathways for proton diffusion, it allows for rapid proton transport. Additionally, it can endure extremely powerful, pulsed electric fields. Onen argues that this is crucial because increasing the device’s voltage enables protons to flow at dizzying speeds.
“The speed certainly was surprising. Normally, we would not apply such extreme fields across devices, in order to not turn them into ash. But instead, protons ended up shuttling at immense speeds across the device stack, specifically a million times faster compared to what we had before. And this movement doesn’t damage anything, thanks to the small size and low mass of protons. It is almost like teleporting,” he says.
“The nanosecond timescale means we are close to the ballistic or even quantum tunneling regime for the proton, under such an extreme field,” adds Li.
Because the protons don’t damage the material, the resistor can run for millions of cycles without breaking down. This new electrolyte enabled a programmable protonic resistor that is a million times faster than their previous device and can operate effectively at room temperature, which is important for incorporating it into computing hardware.
Thanks to the insulating properties of PSG, almost no electric current passes through the material as protons move. This makes the device extremely energy efficient, Onen adds.
Now that they have demonstrated the effectiveness of these programmable resistors, the researchers plan to reengineer them for high-volume manufacturing, says del Alamo. After that, they can scale up the resistor arrays for use in systems and analyze their properties.
They also intend to research the materials in order to get rid of any obstructions that prevent the voltage needed to effectively transfer protons into, through, and out of the electrolyte.
“Another exciting direction that these ionic devices can enable is energy efficient hardware to emulate the neural circuits and synaptic plasticity rules that are deduced in neuroscience, beyond analog deep neural networks,” adds Yildiz
“The collaboration that we have is going to be essential to innovate in the future. The path forward is still going to be very challenging, but at the same time it is very exciting,” del Alamo says.
This research is funded, in part, by the MIT-IBM Watson AI Lab.
