Curve fitting & regression statistics

A raw-data calibration's Step 3 (Analysis) turns your (reference, measured) data points into a model via one of three Calibration methods, then reports a set of standard regression statistics describing how good that model is.

Calibration method

• Polynomial Fit (default) — fits a polynomial curve via least-squares, described below.
• Lookup Table — no curve fit at all. The entered points are the model: evaluating it at any measured value linearly interpolates between the two nearest entered points. Since the model reproduces every entered point exactly by construction, R², RMSE, max error, non-linearity, the uncertainty budget, and the conformity check are all skipped — none of them are meaningful against residuals that are trivially zero at every entered point. Instead, Open Gauge reports linearity deviation — see below. Before building the model, entered points whose reference values are within 1% of the calibration's own reference span of each other are treated as repeat readings of the same test point and collapsed into a single row (reference and measured both averaged) — a delivered lookup table lists one measured value per reference, not several. The final table is always shown with reference values in ascending order.
• Custom Formula — write a formula shape with free parameters, e.g. a*x*sin(x) + b (everything besides x and the allowed function names is treated as a parameter to fit — see Custom formula syntax), and Open Gauge fits those parameters to your data via nonlinear least squares (scipy.optimize.curve_fit), every parameter's initial guess starting at 1.0. R², RMSE, max error, and non-linearity are all computed normally, same as a polynomial fit — it's a real fit, just against a formula shape you chose instead of a fixed polynomial basis. If the fit doesn't converge (common for exponentials/trig functions against badly-scaled data), Open Gauge shows an error instead of a wrong or misleading result.

model_direct (a lab-supplied model with no raw data at all — see Alternative calibration data-entry modes) uses the same Custom Formula mechanism for its own formula input, just with declared parameter values instead of a fit.

The fit

This section covers Polynomial Fit specifically. Open Gauge fits:

$$\text{reference} = f(\text{measured}) = c_0 \cdot \text{measured}^n + c_1 \cdot \text{measured}^{n-1} + \dots + c_n$$

via least-squares (np.polyfit), degree 1 through 5. By default the degree is auto-selected via AIC (Akaike Information Criterion) with a parsimony rule: Open Gauge picks the lowest degree whose AIC isn't beaten by more than 2 points by a higher degree, so it won't fit unnecessary higher-order wiggle to noise. You can also pin a specific degree explicitly in the wizard.

Coefficient covariance. When there are more data points than fitted parameters, Open Gauge also computes the fitted coefficients' covariance matrix (np.polyfit(..., cov=True)), stored alongside the calibration. This matters because evaluating the curve at a point using two or more fitted coefficients together — without their covariance — understates the true uncertainty (GUM Annex H.3 shows 30%+ underestimates from dropping it). Open Gauge stores this matrix so a correct propagation is possible; there isn't yet an Open Gauge feature that evaluates a saved calibration's curve at a live reading, but the covariance is captured for when there is.

A degenerate fit — as few or fewer data points than fitted parameters (e.g. a 2-point straight-line fit) — has zero residual degrees of freedom: no covariance matrix is stored, and the fit-residual uncertainty term (see The uncertainty budget) is marked as having no finite degrees of freedom.

Residuals

For each point, the residual is:

$$\text{residual}_i = \text{reference}_i - f(\text{measured}_i)$$

R²

The coefficient of determination — how much of the variance in the reference values the fitted curve explains. 1.0 is a perfect fit.

RMSE

Root mean square error — the typical magnitude of the residuals:

$$\text{RMSE} = \sqrt{\frac{1}{n}\sum_{i=1}^n \text{residual}_i^2}$$

Max error

The single largest absolute residual across all data points. This is the number Open Gauge compares against a spec when deciding pass/fail — see Decision rules.

%FS error

Max error expressed as a percentage of the measurement span:

$$\%\text{FS error} = \frac{\text{max error}}{\text{range max} - \text{range min}} \times 100\%$$

Non-linearity

For a fit of degree ≥ 2: how far the fitted curve deviates from its own best-fit straight line, as a percentage of full scale. A large non-linearity is a signal that a higher-degree fit was the right call — not just noise the auto-degree-selection happened to pick up.

Linearity deviation

Lookup Table only. Since a lookup table has no fitted curve, non-linearity (which needs one) doesn't apply — but the piecewise-linear interpolation itself introduces its own error between entered points, worth quantifying. Open Gauge fits a straight line (ordinary least-squares) through the calibration's own points, then samples the interpolated curve densely across the measured range and reports how far it strays from that line at each sample:

$$\text{deviation}(x) = \text{LUT}(x) - (\text{slope} \cdot x + \text{intercept})$$

Shown as a chart under the interpolation curve — measured value on the x-axis, deviation on the y-axis, toggleable between absolute (in reference units) and relative (as a percentage of the reference span, %FS) — rather than a single summary number, since where along the range the interpolation strays matters as much as how far. Round markers sit at the calibration's own entered points, connected by the curve, the same visual language as the main calibration chart. It's diagnostic only: Lookup Table has no conformity check (see Calibration method above), so linearity deviation isn't compared against a tolerance or folded into a pass/fail decision.

Repeatability

The standard deviation of measured values at repeated reference points (three or more readings at the same reference value):

$$\text{repeatability} = \sqrt{\frac{1}{m-1}\sum_{j=1}^m (x_j - \bar{x})^2}$$

where $x_1, \ldots, x_m$ are the measured values recorded at the same reference value. Requires at least 3 points at that reference to compute — Open Gauge only reports it when the data supports it.

Hysteresis

The largest spread of measured values recorded at the same reference value across an ascending and descending sweep — i.e. how far apart the "coming up to X" reading and the "coming down to X" reading are, for whichever reference value shows the biggest gap. Requires an up/down sweep in the data (the same reference value visited from both directions) to compute.

See Worked examples for a load-cell example that computes both repeatability and hysteresis from a real 12-point ascending/descending dataset by hand.