On 2026-11-16 7:22 p.m., in response to some suggestions from Bob Worcester, I wrote:
I think the 3b1b series (starting with https://www.youtube.com/watch?v=aircAruvnKk) is well worth digging into. The second link you sent (at https://youtu.be/olDwTWN9vs4?si=NYKdlvasO6nOhA0N ) is an interesting example of something, but I think is wrong to identify the iterations of Newton’s method as training runs of a learning system. And this last one (at https://www.youtube.com/watch?v=jmmW0F0biz0&t=10s )is I think “less complicated” only because it explains so much less (and overlooks the importance, explained in the second one, of NON-linear behaviour of a neuron).
On 2026-11-16 11:41 a.m., Robert Worcester wrote:
This is a less complicated explanation of neural networks: https://www.youtube.com/watch?v=jmmW0F0biz0&t=10s
On 2026-11-12 8:34 a.m., Robert Worcester wrote:
The best simple explanation I have run across so far.
https://youtu.be/olDwTWN9vs4?si=NYKdlvasO6nOhA0N
The math is (to me) scary BUT the organic structure that this machinery seek to emulate is MUCH more complicated than the math. Where does physics become biology (or vise versa)?
On Wed, 11 Nov 2026 at 07:58, Robert Worcester wrote:
Hi – In response to our discussions yesterday I was motivated to try to better “understand” how neural networks (wet or dry) actually work.
I found this helpful though I confess I don’t really understand the math of linear algebra BUT it looks potentially “understandable.”
I am wondering if a mathematician can facilitate the transmission of the neural weightings necessary for such understanding?
A Simple Model Neuron is something that accepts as input (from a set of sources which we’ll call its “dendritic contacts” or “upstream layer”) a discrete set of signal strengths (each of which can be represented by a real number), and for each such collection of inputs sends to a destination (which we’ll call its “downstream channel” or “axon”) an output signal (of the same kind as the inputs). But in order to be useful for learning, it must ALSO accept inputs that it interprets as requests to increase or decrease its output value, to which it responds with some combination of adjusting its own behaviour and/or passing on requests for changes of its other inputs. Such requests coming, either directly or indirectly, from the output channel are referred to as “back propagation”. This can be done in various ways, and the details may be hard to follow, so it is often left out of discussions of how to model a single neuron.
For example, the model of a single neuron from 1:05 to 3:15 in the video by Aviraj Ingle deals well enough with the forward process but omits any reference to the adjustment capability which is required for “back propagation” in neural networks. [This is followed by a section in which oddly combines discussion of the need for nonlinearity in order for network “layers” to be useful with how cutoffs enhance the capabilities of a single layer, and then an example in which back propagation is actually used without ever having been properly explained and in which iteration of steps in producing the response to a single data set is confounded with the process of training with different sample data sets.]
A common form for the forward propagation through a single neuron just takes a weighted sum of the inputs giving result $#r=\Sigma w_{i} x_{i}#$ and then gives (or perhaps just goes towards) $#0#$ for results $#r#$ that are smaller than some lower cutoff $#m#$, and $#1#$ for all that are larger than some upper cutoff $#M#$, and scales intermediate results between $#m#$ and $#M#$ into outputs between $#0#$ and $#1#$. [We can do this in many ways – eg with strict cutoffs and linear scaling to get $#y=(\frac{1}{2}+\frac{|r-m|-|r-M|}{2(M-m)})#$, or more smoothly with a “sigmoid” $#y=\sigma(z)=(\frac{1}{1+e^{-z}})#$ with $#z=(r-\frac{m+M}{2})(\frac{4}{M-m})#$]
Even without back-propagation, if we choose the weights and cutoffs directly then such a neuron does provide a model for making or predicting some simple kinds of decision. For example if someone is in a situation where they have access to a down jacket but no waterproof layer and we want to predict the probability of whether or not they will go out on the basis of the predicted probabilities of cold and wet weather, then we might assign a high negative weight to the probability of precipitation and a low negative weight to the probability of having a temperature below 10 Celsius. But if we expect that they have some desire to actually go out then we might have a lower cutoff corresponding to how strong we expect that desire to be and the probability that they actually decide to go out might then be equal to the output from such a neuron with inputs equal to the applicable probabilities of rain and cold. Or in biology, such a fixed-weight neuron might enable a simple organism to decide whether or not to move on the basis of its sensitivities to the concentrations of various offensive chemicals in its environment with a cutoff based on its need not to waste energy in unnecessary motion. But in such cases the neuron just has fixed behaviour – either evolved over many generations or put in by a human, and there is no immediate learning involved.
It may seem that the use of linear functions is unduly restrictive, but with the inclusion of cutoffs a bunch of such neurons acting in parallel can approximately model any smooth function.
For example, the function $#f(x)=1-x^2#$ for $#-1<x<1#$ can be approximated to within $#\pm0.1#$ by the piecewise linear function $#f(x)=2(x+1) for -1<x<-3/4 ; 1+2x/3 for -3/4<x<0; 1-2x/3 for 0<x<3/4; and -2(x-1) for 3/4<x<1#$ which is just the sum of the outputs of four single input neurons. But here the neurons have been given their weights by me and there was no learning involved.
Now let’s re-introduce the learning process.
Rather than put in the actual weights and cutoffs we can start by putting in random values and let the neurons themselves figure out what to do. But for this we need them to be able to detect where they went wrong and have some way they can arrange to do better.
Here’s an example with two inputs. If we call the input values $#x_1#$ and $#x_2#$ and the output is initially given by $#y=w_1x_1+w_2x_2#$, then a request to increase the output can be met by adjusting the “weights” $#w_1#$ and $#w_2#$ in various ways but for fixed input values the fastest increase per unit “distance” in the $#(w_1,w_2)#$ space is when the change $#(\Delta w_1,\Delta w_2)#$ is parallel to the input vector $#(x_1,x_2)#$. [And also, if we are able to change the inputs, then for any given values of $#w_1#$ and $#w_2#$ the output can be increased by requesting an increment of the input values for which the most efficient choice is when $#(\Delta x_1,\Delta x_2)#$ is parallel to the weights vector $#(w_1,w_2)#$.]



