What is a Neuron?

When people talk about neurons in machine learning, they usually mean a tiny mathematical unit that takes some numbers in and produces a single number out.

At its core, a neuron does three things:

1. Takes inputs – a vector of numbers, e.g. features from your data.
2. Applies weights and a bias – it computes a weighted sum of those inputs.
3. Passes the result through an activation function – often something like ReLU.

Weights and bias

Suppose your inputs are $x_1, x_2, \dots, x_n$.
Your neuron has the same number of weights $w_1, w_2, \dots, w_n$ and one bias $b$.

It first computes:

What does this mean?

• $x_1, x_2, \dots, x_n$ are your inputs (the data you feed in, like pixel values or features).
• $w_1, w_2, \dots, w_n$ are the weights (how much each input matters).
• $b$ is the bias (a constant that shifts everything).
• $z$ is the result (a single number).

A simple example: If you have 3 inputs with values $x_1 = 2$, $x_2 = 3$, $x_3 = 1$, and weights $w_1 = 0.5$, $w_2 = 0.3$, $w_3 = 0.8$, and bias $b = 1$, then:

• Weights tell the neuron how important each input is. A larger weight means that input has more influence.
• The bias shifts the whole function left or right, letting the neuron fire even when all inputs are small or zero.

You can think of $z$ as the raw score before the neuron decides how strongly to respond.

ReLU and other activations

If we only used the weighted sum $z$, a whole network of neurons would just behave like one big linear function. That’s why we add a non-linear activation function.

One of the most popular is ReLU (Rectified Linear Unit):

What does this mean?

$\max(0, z)$ means “take the larger of 0 and $z$”. In other words:

• If $z$ is positive (like $z = 5$), ReLU outputs that same number: $\text{ReLU}(5) = 5$.
• If $z$ is negative (like $z = -3$), ReLU outputs 0: $\text{ReLU}(-3) = 0$.

Examples:

• $\text{ReLU}(7) = 7$ (positive stays positive)
• $\text{ReLU}(-2) = 0$ (negative becomes zero)
• $\text{ReLU}(0) = 0$ (zero stays zero)

This simple shape makes networks easier to train and gives them the ability to model complex, non-linear relationships in data. It’s like a “rectifier” that cuts off anything below zero.

Other activation functions

ReLU is just one choice. A few other common activations:

• Sigmoid – squashes values into $(0, 1)$, historically used in early neural nets and for probabilities:

  $$\sigma(z) = \frac{1}{1 + e^{-z}}$$

  What does this mean?

  • $e$ is Euler’s number (approximately 2.718), a mathematical constant.
  • $e^{-z}$ means “e raised to the power of negative z”.
  • The formula takes any number $z$ (positive or negative) and squashes it into a value between 0 and 1.

  Examples:

  • If $z = 0$, then $\sigma(0) = \frac{1}{1 + e^0} = \frac{1}{1 + 1} = 0.5$ (middle value).
  • If $z$ is very large (like $z = 10$), $\sigma(10) \approx 1$ (approaches 1).
  • If $z$ is very negative (like $z = -10$), $\sigma(-10) \approx 0$ (approaches 0).

  Think of it as a smooth S-curve that converts any number into a probability-like value between 0 and 1.

• Tanh – similar to sigmoid but outputs in $(-1, 1)$ and is zero-centered:

  $$\tanh(z) = \frac{e^{z} - e^{-z}}{e^{z} + e^{-z}}$$

  What does this mean?

  • Like sigmoid, this uses $e$ (Euler’s number).
  • $e^z$ is “e raised to the power of z”, and $e^{-z}$ is “e raised to the power of negative z”.
  • The formula divides the difference by the sum, which creates an S-curve like sigmoid, but centered at zero.

  Examples:

  • If $z = 0$, then $\tanh(0) = 0$ (exactly zero, unlike sigmoid which gives 0.5).
  • If $z$ is large positive (like $z = 5$), $\tanh(5) \approx 1$ (approaches 1).
  • If $z$ is large negative (like $z = -5$), $\tanh(-5) \approx -1$ (approaches -1).

  The key difference from sigmoid: tanh outputs can be negative, and it’s centered at zero, which can help with training.

• Leaky ReLU – like ReLU but allows a small negative slope so neurons don’t die completely:

  $$\text{LeakyReLU}(z) = \begin{cases} z & \text{if } z \ge 0 \\ \alpha z & \text{if } z < 0 \end{cases}$$

  What does this mean?

  This is a piecewise function (it behaves differently depending on the value of $z$):

  • If $z \ge 0$ (zero or positive): output is just $z$, same as regular ReLU.
  • If $z < 0$ (negative): output is $\alpha \times z$, where $\alpha$ is a small number like $0.01$.

  Examples:

  • If $z = 5$, then $\text{LeakyReLU}(5) = 5$ (positive stays positive).
  • If $z = -3$, then $\text{LeakyReLU}(-3) = 0.01 \times (-3) = -0.03$ (small negative value, not zero).

  The $\alpha$ parameter (often $0.01$) is the “leak” – it lets a tiny bit of negative signal through instead of completely cutting it off. This prevents neurons from “dying” (always outputting zero) during training.

Putting it together

A single neuron computes:

What does this mean?

This is the complete formula for a neuron:

1. First part ($w_1 x_1 + \dots + w_n x_n + b$): Compute the weighted sum of inputs plus bias. This gives you $z$ (the raw score).
2. Second part ($\phi(\dots)$): Apply the activation function to $z$. The Greek letter $\phi$ (phi) is just a placeholder for whatever activation function you choose.

In plain English: Multiply each input by its weight, add them all up, add the bias, then pass that result through an activation function.

Example with ReLU:

• Inputs: $x_1 = 2$, $x_2 = 3$
• Weights: $w_1 = 0.5$, $w_2 = 0.3$
• Bias: $b = 1$
• First compute: $z = (0.5 \times 2) + (0.3 \times 3) + 1 = 2.9$
• Then apply ReLU: $\text{output} = \text{ReLU}(2.9) = 2.9$

where $\phi$ can be ReLU, sigmoid, tanh, or something more modern like GELU or SiLU.

Comparing activation functions

When you stack many neurons into layers, and many layers into a network, you get the neural networks used in modern deep learning. But underneath, it’s all built from this same pattern: inputs → weighted sum + bias → activation.

This post is my starting point while I’m learning more about how these pieces connect to training, loss functions, and backpropagation.