DiscoverScroll article

In machine learning, a neural network (NN) or neural net, is a computational model inspired by the structure and functions of biological neural networks.

A neural network consists of connected units or nodes called artificial neurons, which loosely model the neurons in the brain. The "signal" is a real number, and the output of each neuron is computed by some non-linear function of the totality of its inputs, called the activation function. Signals travel from the first layer (the input layer) to the last layer (the output layer), typically passing through multiple intermediate layers (hidden layers). A network is typically called a deep neural network if it has at least two hidden layers. Deep neural networks are capable of learning sophisticated hierarchical representations. Training neural networks is a compute-intensive process, accelerated by the use of graphics processing units (GPUs), and large datasets. Architectural innovations such as convolutional neural networks (CNNs) significantly improved performance in computer vision tasks, while recurrent neural networks (RNNs) enabled modeling of sequential data such as speech and time-series information. Transformer architectures introduced attention mechanisms that allow neural networks to model long-range dependencies in data and have been the basis of large language models. Artificial neural networks are used for a myriad of tasks including chatbots, large-scale text, image, and video generation, and robotics.

The simplest kind of feedforward neural network (FNN) is a linear network, which consists of a single layer of output nodes with linear activation functions; the inputs are fed directly to the outputs via weights.

Neural networks instead originated from efforts to model information processing in biological systems via connectionism. Hebb proposed a learning hypothesis based on neural plasticity that became known as Hebbian learning. It was used in many early neural network experiments, such as Rosenblatt's perceptron and the Hopfield network.

Interest in neural networks revived during the 1980s because of the novel backpropagation algorithm, which allowed multi-layer neural networks to be trained efficiently by propagating error gradients backward (from output back to input) through network layers.

Deep learning architectures for convolutional neural networks (CNNs) with convolutional layers and downsampling layers and weight replication began with the neocognitron introduced by Kunihiko Fukushima in 1979.

In 1991, Jürgen Schmidhuber proposed the "neural sequence chunker" or "neural history compressor" which introduced self-supervised pre-training (the "P" in ChatGPT) and neural knowledge distillation.

In 2014, the state of the art was training " very deep neural network" with 20 to 30 layers. In 2015, training very deep networks advanced with the highway network, published in May, and the residual neural network (ResNet) in December.

Machine learning has involved a variety of approaches to training models, including supervised learning, unsupervised learning, reinforcement learning, and self-supervised learning.

Stochastic neural networks originating from Sherrington–Kirkpatrick models are a type of neural network built by introducing random variations into the network, either by giving neurons stochastic transfer functions, or by giving them stochastic weights. Stochastic neural networks trained using a Bayesian approach are known as Bayesian neural networks.

Neural architecture search (NAS) uses machine learning to automate NN design.

By adopting a softmax activation function, a generalization of the logistic function, on the output layer of the neural network (or a softmax component in a component-based network) for categorical target variables, the outputs can be interpreted as posterior probabilities.

Neuromorphic engineering or a physical neural network constructed non-von-Neumann chips to directly implement neural networks in circuitry.

These concerns have led to increased research in explainable artificial intelligence (XAI), robust machine learning, and hybrid AI approaches that combine neural learning with symbolic reasoning.

Next Generation of Neural Networks Archived 24 January 2011 at the Wayback Machine – Google Tech Talks

Neural Networks and Information Archived 9 July 2009 at the Wayback Machine

Quick Facts

• Machine learning has involved a variety of approaches to training models, including supervised learning, unsupervised learning, reinforcement learning, and self-supervised learning.
• A network is typically called a deep neural network if it has at least two hidden layers.
• A neural network consists of connected units or nodes called artificial neurons, which loosely model the neurons in the brain.
• Architectural innovations such as convolutional neural networks (CNNs) significantly improved performance in computer vision tasks, while recurrent neural networks (RNNs) enabled modeling of sequential data such as speech and time-series information.
• Deep neural networks are capable of learning sophisticated hierarchical representations.

Source material: Wikipedia - "Neural network (machine learning)". Adapted and summarized for DiscoverScroll. Original contributors are credited through the linked Wikipedia article. Read original on Wikipedia. CC BY-SA 4.0. Changes were made from the original.