← All articles

Machine learning · Optimization · Reto 4

Inside logistic regression

Follow a straight decision boundary into a convex optimization problem. Explore feature vectors, probabilities, binary cross-entropy, BFGS, and the geometry of the Hessian.

On this page

Two features and a straight line are enough to build the classifier in my Reto 4 notebook. Finding that line takes us from dot products in feature space to a convex loss in parameter space. This field guide follows that path in 11 sections, pairing the original notebook figures and results with browser-based experiments.

The interactive labs run in your browser with JavaScript and local assets.

The classification problem

00 / ORIGINAL NOTEBOOK

The notebook trains a binary classifier from radius_mean and smoothness_mean to predict pronostic: 1 for malignant, 0 for non-malignant. It optimizes probabilities, then evaluates the resulting linear decision boundary.

Training observations
392
180 class 0 · 212 class 1
Original → final loss
0.6931 → 0.2122
Binary cross-entropy · 20 BFGS iterations
Training accuracy
91.58%
359 correct · 33 errors
w∗=(5.171764661.604467940.56221445)w^* = \begin{pmatrix} 5.17176466 \\ 1.60446794 \\ 0.56221445 \end{pmatrix}
raw observations
392 × 2
→
add bias column
Z: 392 × 3
→
scores & probabilities
→
cross-entropy f(w)
→
minimize over w ∈ ℝ³
Source: Reto4LuisOtalora.html. These numerical results and the original figures below come from the Reto 4 notebook. The interactive point-cloud and optimizer experiments later in this guide use a small explicitly labeled teaching dataset; the 392 original CSV rows are not embedded in the notebook HTML.
Original scatter plot of the 392 observations in the notebook
Original notebook: the training observations in feature space.
Original notebook plot of loss decreasing across solver iterations
Original notebook: logged loss from the initial vector through the 20 solver iterations.

The whole problem is a dot product, repeated 392 times

01 / LINEAR ALGEBRA

Each observation begins as a 2D feature vector x=(x1,x2)Tx = (x_1, x_2)^{\mathsf T}. The notebook appends a constant 1, producing a 3D augmented vector z=(x1,x2,1)Tz = (x_1, x_2, 1)^{\mathsf T}. That extra coordinate lets the model learn an intercept without a separate operation.

One observation

zi=(x1ix2i1),w=(w1w2w3)si=wTzi=w1x1i+w2x2i+w3\begin{aligned} z_i &= \begin{pmatrix} x_{1i} \\ x_{2i} \\ 1 \end{pmatrix},\quad w = \begin{pmatrix} w_1 \\ w_2 \\ w_3 \end{pmatrix} \\ s_i &= w^{\mathsf T}z_i \\ &= w_1 x_{1i} + w_2 x_{2i} + w_3 \end{aligned}

The output sᵢ is a scalar called a logit or score. Geometrically, w₁ and w₂ define the orientation of a line; w₃ translates it.

All observations simultaneously

Z∈R392×3w∈R3,s∈R392Zw⟶392 logitsσ(Zw)⟶392 probabilitiesZT(y−t)∈R3\begin{aligned} Z &\in \mathbb{R}^{392\times 3} \\ w &\in \mathbb{R}^3,\quad s \in \mathbb{R}^{392} \\ Zw &\longrightarrow 392\text{ logits} \\ \sigma(Zw) &\longrightarrow 392\text{ probabilities} \\ Z^{\mathsf T}(y-t) &\in \mathbb{R}^3 \end{aligned}

Each row of Z is one observation. Each column is a model input (two features and a constant).

Try a matrix multiplication by handUse the sliders to change one augmented row
Bias coordinate 1 (fixed)
A 1 multiplies the intercept w₃.
s=5.1718(0.70)+1.6045(−0.80)+0.5622≈2.89888\begin{aligned} s &= 5.1718(0.70) + 1.6045(-0.80) + 0.5622 \\ &\approx 2.89888 \end{aligned}
Logit s
—
Probability σ(s)
—
Prediction ≥ 0.5
—

This is a synthetic example feature vector evaluated with the actual optimized weights. Its computed probability is illustrative, not a record taken from the original dataset.

Why matrix notation matters: NumPy can calculate every score with a single Z @ w. More importantly, derivatives become compact: ∇f(w)=ZT(y−t)m\nabla f(w) = \frac{Z^{\mathsf T}(y-t)}{m} and H(w)=ZTDZmH(w) = \frac{Z^{\mathsf T}DZ}{m}. Both are built from the same matrix Z.

Three coefficients determine one straight line

02 / FEATURE SPACE

Logistic regression is called “regression”, but the notebook uses it for binary classification. First compute s=w1x1+w2x2+w3s = w_1 x_1 + w_2 x_2 + w_3; then turn it into p=σ(s)=11+e−sp = \sigma(s) = \frac{1}{1 + e^{-s}}. The class threshold of 0.5 corresponds exactly to s = 0.

w1x1+w2x2+w3=0x2=−w1w2x1−w3w2\begin{aligned} w_1 x_1 + w_2 x_2 + w_3 &= 0 \\ x_2 &= -\frac{w_1}{w_2}x_1 - \frac{w_3}{w_2} \end{aligned}
Interactive decision boundaryTeaching dataset · not the original 392 records
Illustrative class 0Illustrative class 1Selected example x = (0.7, −0.8)
Boundary slope
—−w₁/w₂
Boundary intercept
—−w₃/w₂
Example prediction
—p(class 1) at blue point

Two different spaces, never confuse them: the x₁–x₂ chart shows patients/features and a decision line; the w₁–w₂–w₃ space contains all possible models, and each point in that space has a loss. The loss has a convex geometry there—even though the decision boundary in feature space is linear.

Why does the notebook minimize cross-entropy?

03 / PROBABILITY & LOSS

We want high probability on the true class, not just the right side of a threshold. With a Bernoulli model, the likelihood of label t∈{0,1}t \in \{0,1\} is pt(1−p)1−tp^t (1-p)^{1-t}. Minimizing negative log-likelihood gives the binary cross-entropy (BCE) used in the notebook.

ℓ(p,t)=−tlog⁡p−(1−t)log⁡(1−p)\ell(p,t) = -t\log p - (1-t)\log(1-p)
One prediction, two loss choicesCompare BCE vs squared probability error

Sigmoid: turns a logit into a probability

Loss as a function of the logit s

Predicted p(class 1)
—
BCE contribution
—optimized by the notebook
Squared error
—comparison only

BCE has the right probabilistic interpretation

Its average is the negative log-likelihood of the training labels, so minimizing BCE is maximum likelihood estimation for Bernoulli observations. It heavily penalizes confident wrong predictions and rewards calibrated probabilities.

Why not optimize accuracy directly?

Accuracy jumps only when a prediction crosses 0.5. As a function of w, it is a discontinuous, piecewise-constant objective with no useful ordinary gradient. BCE is smooth, differentiable and convex in w for this linear-logit model.

# Original stable implementation s = Z @ w loss = np.mean(np.logaddexp(0.0, s) - t * s)

Why np.logaddexp? It computes log⁡(1+es)\log(1+e^s) stably even for very large positive s. Algebraically, ℓ(s,t)=log⁡(1+es)−ts\ell(s,t) = \log(1+e^s) - ts is exactly the same BCE, expressed in logits. A direct np.log(1+np.exp(s)) may overflow. The notebook stabilizes its loss; for very extreme logits the standalone sigmoid could also be implemented more robustly with scipy.special.expit.

Why is the loss a convex curve, and why is that powerful?

04 / CONVEX ANALYSIS

A convex function lies at or below the straight chord connecting any two points on its graph. For a twice differentiable one-variable function, a nonnegative second derivative everywhere is enough to prove convexity. Logistic cross-entropy satisfies that condition as a function of the logit.

ℓ′′(s,t)=σ(s)(1−σ(s))≥0\ell^{\prime\prime}(s,t) = \sigma(s)\bigl(1 - \sigma(s)\bigr) \geq 0
Move the chord: visualize Jensen's inequalityExperiment with BCE and an alternative
Curve value f(mixture)
—
Chord value
—
Jensen inequality
—Convex if f(mix) ≤ chord for all pairs

From a convex curve to a convex surface

Every logit si=ziTws_i = z_i^{\mathsf T} w is affine in w. A convex loss composed with an affine function remains convex. An average of convex functions remains convex. Therefore, f(w) is convex throughout ℝ³.

Why the sigmoid isn't the convex object

The sigmoid is S-shaped: it changes curvature. Its graph is not globally convex. What is convex is cross-entropy applied to the linear score as a function of the parameters. Squared error applied to sigmoid probabilities generally loses this guarantee.

Global-minimum guarantee: For a differentiable convex f, any stationary point (∇f(w)=0\nabla f(w) = 0) is a global minimizer. With the notebook's full-rank design matrix and finite weights, the logistic Hessian is positive definite, making the objective strictly convex and its finite minimizer unique. Convexity does not mean BFGS magically reaches the solution in one step, or that the training loss becomes zero.

What exactly happens inside minimize?

05 / THE OPTIMIZER

The notebook calls minimize(perdida, w0, args=(Z, t), callback=guardar_perdida). Because there are no bounds, constraints, Jacobian or method specified, SciPy chooses BFGS by default for this unconstrained problem, numerically approximates derivatives and iteratively improves w.

Gradient descent: useful baseline

wk+1=wk−α∇f(wk)w_{k+1} = w_k - \alpha\nabla f(w_k)

It follows the local steepest-descent direction. The step-size α affects stability: too small is slow; too large can overshoot.

BFGS: the notebook's method

pk≈−Bk∇f(wk)wk+1=wk+αkpk\begin{aligned} p_k &\approx -B_k\nabla f(w_k) \\ w_{k+1} &= w_k + \alpha_k p_k \end{aligned}

Bₖ is a changing approximation of the inverse Hessian. BFGS adapts to curvature and uses a line search. It does not require you to supply the true Hessian.

Start
w₀ = (0,0,0)
→
Evaluate
f(wₖ)
→
Estimate
∇f(wₖ)
→
Find direction
and line-search step
→
Update w,
check stop
Optimizer sandbox: run the iterations yourselfToy logistic dataset · true computed losses, not historical notebook steps
Iterations so far
0
Current toy loss
—
Gradient magnitude
—
Current w
—

Important: This executes a minimal gradient-descent or inverse-Hessian BFGS demonstration in JavaScript on deterministic toy observations. SciPy's production BFGS has a more sophisticated line search, stopping conditions and derivative approximations. The notebook's actual run is still the separate 20-iteration plot above.

What is saved by the callback?

The notebook initializes historial_f = [perdida(w0,Z,t)] manually and appends a loss each time the callback receives an updated vector. callback is for observation/recording; it does not change the mathematical objective.

Why did it stop?

Its output reports success=True, 20 iterations, f(w∗)=0.212163898f(w^*) = 0.212163898, and ∥∇f(w∗)∥≈4.16×10−6\|\nabla f(w^*)\| \approx 4.16 \times 10^{-6}. A small gradient indicates little first-order improvement remains; combined with convexity, this supports a global minimizer.

The gradient is a direction; the Hessian is a curvature map

06 / DIFFERENTIAL GEOMETRY

The notebook derives the gradient and Hessian analytically for f(w)f(w). The gradient is a 3-vector indicating how sensitive the loss is to each coefficient. The Hessian is a symmetric 3×3 matrix describing how the gradient changes as we move in parameter space.

y=σ(Zw)∈R392∇f(w)=ZT(y−t)392∈R3D=diag⁡(yi(1−yi))H(w)=ZTDZ392∈R3×3\begin{aligned} y &= \sigma(Zw) \in \mathbb{R}^{392} \\ \nabla f(w) &= \frac{Z^{\mathsf T}(y-t)}{392} \in \mathbb{R}^3 \\ D &= \operatorname{diag}\bigl(y_i(1-y_i)\bigr) \\ H(w) &= \frac{Z^{\mathsf T}DZ}{392} \in \mathbb{R}^{3\times 3} \end{aligned}
Proof that Hessian is positive semidefinite: for any v ∈ ℝ³,
vTH(w)v=1392∑i=1392yi(1−yi)(ziTv)2≥0\begin{aligned} v^{\mathsf T} H(w) v &= \frac{1}{392}\sum_{i=1}^{392} y_i(1-y_i)(z_i^{\mathsf T}v)^2 \\ &\geq 0 \end{aligned}
Every term is nonnegative. At finite w, all yᵢ(1−yᵢ) are strictly positive mathematically, and because the notebook reports rank⁡(Z)=3\operatorname{rank}(Z) = 3, this expression is positive for every v ≠ 0. That is the deeper reason for strict convexity here—not merely that three eigenvalues happened to be positive at the final point.
Walk around the convex bowlSecond-order Taylor approximation at the actual notebook optimum

Fix w₃ at its optimized value. Move w₁ and w₂ near w*. Using the actual Hessian matrix, the local loss is approximated by f(w∗+δ)≈f(w∗)+∇f(w∗)Tδ+12δTH∗δf(w^* + \delta) \approx f(w^*) + \nabla f(w^*)^{\mathsf T}\delta + \tfrac12 \delta^{\mathsf T}H^*\delta. The colors and ellipses are not exact global loss contours because the original 392 CSV rows are absent from this HTML.

Approx. local f(w)
—
Approx. gradient norm
—
Δw = (Δw₁,Δw₂,0)
—

Real Hessian at w*

H(w∗)≈(0.012118−0.015377−0.012501−0.0153770.0595020.009306−0.0125010.0093060.065703)H(w^*) \approx \begin{pmatrix} 0.012118 & -0.015377 & -0.012501 \\ -0.015377 & 0.059502 & 0.009306 \\ -0.012501 & 0.009306 & 0.065703 \end{pmatrix}

Off-diagonal terms describe interaction between parameters; symmetry follows from continuous mixed partial derivatives.

Actual Hessian eigenvalues

λ1=0.00603294λ2=0.05323680λ3=0.07805335\begin{aligned} \lambda_1 &= 0.00603294 \\ \lambda_2 &= 0.05323680 \\ \lambda_3 &= 0.07805335 \end{aligned}

All positive → positive definite at w*. Eigenvectors identify principal directions; eigenvalues measure curvature along them. A small eigenvalue means a relatively flat direction.

Remember: ∇f(w∗)≈(−1.06, 3.52, 1.97)×10−6\nabla f(w^*) \approx (-1.06,\, 3.52,\, 1.97) \times 10^{-6} is small but not exactly zero because this is numerical optimization. The Hessian is not expected to be zero at the minimum; a positive Hessian is the signature of an upward-curving valley.

Why loss and classification metrics answer different questions

07 / EVALUATION

Loss evaluates the quality of predicted probabilities. A confusion matrix evaluates hard decisions after thresholding at 0.5. The notebook reports both on the same training observations, so these are training results—not evidence of generalization to unseen patients.

Explore the original confusion matrixClick any cell to inspect its meaning
Predicted 0
Predicted 1
Real 0
Real 1

Choose a cell

The diagonal contains correct classifications; the off-diagonal cells contain errors.

Accuracy91.58%
Precision (class 1)93.66%
Recall / sensitivity90.57%
Specificity92.78%
F1 score92.09%
Accuracy=TP+TNm=192+167392≈91.58%Precision=TPTP+FP=192205≈93.66%Recall=TPTP+FN=192212≈90.57%\begin{aligned} \text{Accuracy} &= \frac{TP+TN}{m} \\ &= \frac{192+167}{392} \approx 91.58\% \\ \text{Precision} &= \frac{TP}{TP+FP} \\ &= \frac{192}{205} \approx 93.66\% \\ \text{Recall} &= \frac{TP}{TP+FN} \\ &= \frac{192}{212} \approx 90.57\% \end{aligned}

What the notebook truly demonstrates

BFGS finds a lower training BCE than the prior candidates. The optimized model makes 359 correct hard predictions on the 392 training rows, at the notebook's fixed threshold.

What you would test next

Use held-out test data or cross-validation; examine ROC-AUC, precision–recall trade-offs and calibration, and choose the decision threshold based on the cost of each error. None of those tests were carried out in the Reto 4 notebook.

Which candidate wins, and by what definition?

08 / THE FOUR MODELS

The notebook compares three fixed candidates from Reto 3 with the BFGS solution from Reto 4. They are evaluated on the same Z and t, so their losses are directly comparable.

Original notebook valuesSmaller cross-entropy and gradient norm are preferable here
Optimized
0.212164
a
0.221731
c
0.393056
b
0.653523
CandidateWeights (w₁,w₂,w₃)‖∇f(w)‖Smallest λ(H)
Optimized(5.1718, 1.6045, 0.5622)0.0000040.006033
a(4.0364, 1.3192, 0.0603)0.0274940.011861
c(1.0000, 0.5020, 0.0603)0.1590990.134484
b(1.3450, 3.5970, 0.5880)0.2451930.014797

The optimized loss is about 4.31% lower than candidate a. Every reported Hessian in this comparison is positive definite at the evaluated point. This does not imply all four have equal fit or gradients.

Original notebook graph comparing candidate boundaries with optimized boundary
Original notebook: dashed a/b/c decision boundaries versus the optimized boundary, over the real scatter plot.

Can you reason through the whole chain?

09 / CONCEPT CHECKS

Try to answer each question before expanding it. Each answer is tied to a specific idea used by the notebook.

Why does Z have 3 columns if there are only 2 features?
The third column consists of ones. It multiplies the intercept w₃, making every logit a dot product between two 3-vectors: [x₁,x₂,1] and [w₁,w₂,w₃].
If a point is classified correctly, does its loss have to be small?
Not necessarily. A correct class-1 probability of 0.51 has BCE ≈ 0.673, while a confident correct prediction of 0.99 has BCE ≈ 0.010. Both count as correct under accuracy, but BCE distinguishes them.
The sigmoid curve has an S shape. How can logistic regression be convex?
The sigmoid itself is not globally convex. The objective is BCE of an affine logit: softplus(zᵀw) − t(zᵀw). Its Hessian ZᵀDZ/m is positive semidefinite for every finite parameter vector.
Why does a positive definite Hessian at the solution matter?
It indicates upward curvature in every local parameter direction. Together with the globally convex structure and a very small gradient, it supports a unique global minimizer for the full-rank model. Positive definiteness at just one arbitrary point alone does not prove global convexity for an unrelated function.
Does SciPy's BFGS use the hessiana(w,Z,t) function in this notebook?
No. The notebook passes only perdida, w0, args and a callback. BFGS estimates gradients numerically here and builds its own inverse-Hessian approximation. The analytical Hessian is computed later for evaluation.
Can 91.58% training accuracy be interpreted as clinical accuracy on future patients?
No. This percentage is computed on the same 392 observations used for training. Independent validation or test data and appropriate clinical evaluation are needed before interpreting performance on unseen patients.

The notebook, line by line in mathematical terms

10 / CODE MAP

Core functions, matching the notebook implementation

def sigmoide(s):
    return 1 / (1 + np.exp(-s))
def perdida(w, Z, t):
    s = Z @ np.asarray(w, dtype=float)
    return np.mean(np.logaddexp(0.0, s) - t * s)
def gradiente(w, Z, t):
    y = sigmoide(Z @ w)
    return Z.T @ (y - t) / len(t)
def hessiana(w, Z, t):
    y = sigmoide(Z @ w)
    pesos = y * (1 - y)
    return (Z.T * pesos) @ Z / len(t)
w0 = np.zeros(3)
historial_f = [perdida(w0, Z, t)]
def guardar_perdida(w):
    historial_f.append(perdida(w, Z, t))
resultado = minimize(perdida, w0, args=(Z, t),
                     callback=guardar_perdida)
w_opt = resultado.x

What each object means

resultado.x is the parameter vector; resultado.fun is the objective value at that vector; resultado.nit counts optimizer iterations; resultado.success is its success flag. The callback records iterates, but function evaluations can be more numerous than iterations.

Optional clarity improvement

resultado = minimize(
    perdida, w0,
    args=(Z, t),
    method="BFGS",
    jac=gradiente,
    callback=guardar_perdida
)

This is an additional suggestion, not the original code. Passing jac=gradiente provides the analytical gradient, avoiding BFGS's need to estimate it by finite differences. It can change numerical details of the run.

Final mental model: rows of Z are observations → Z @ w forms linear scores → sigmoid gives probabilities → BCE measures mismatch → BFGS searches in parameter space → gradient indicates where to move → Hessian shows the bowl’s curvature → convexity makes a stationary point globally optimal → the resulting w defines a straight classification boundary in feature space.

Source: Reto 4 (Reto4LuisOtalora.html). The training figures and results come from the notebook. The interactive labs identify their teaching data and local quadratic approximation. The clinical labels are part of an educational dataset.