Scanner ICC Profile Generator Accuracy Benchmark

1. Key metrics (absolute colorimetric · ΔE2000)

Mean covers the overall level; the tail covers consistency. ICC quality should be judged together with the hit rates at ≤1/≤2/≤5, the shape of the error distribution, and whether any patches run away.

2. Ranking under the three rendering intents

Intent tags: A2B0 = perceptual, A2B1 = relative colorimetric, A2B2 = saturation; missing tables fall back to A2B0. Primary metric = absolute colorimetric (same semantics as the reference data).

2.1 Absolute colorimetric (primary metric)

Same semantics as the absolutely measured reference data; accuracy is judged on this metric.

2.2 Relative colorimetric

Media white is mapped to the PCS white point. ArgyllCMS's 3.41 here is not a mapping defect: its wtpt declares the measured paper white (L*≈93.5) and relative colorimetric performs no such white-point rescaling, so the rendering comes out ≈4 L* too bright. The same profile scores 0.72 in absolute colorimetric (ΔL residual 0.00).

2.3 Perceptual

This column always uses A2B0; profiles without A2B1/A2B2 fall back to A2B0 in the relative and saturation columns as well. ArgyllCMS ships only A2B0, so its value here equals its relative colorimetric result (see the note under 2.2).

3. Mean ΔE00 under absolute colorimetric

4. Profile structure

Column notes: "Header intent" is the rendering-intent field at ICC header offset 64 — a vendor-declared default that CMMs ignore when the caller specifies an intent explicitly.

5. Method and reproduction

Test dataset download

6. Scope and caveats

Four statements to read before the tables: they determine what the numbers can and cannot be read as, and they are why this report argues about ranking and order of magnitude rather than about "a 0.05 gap proves superiority".

1. In-sample data: every profile evaluated here — ColorRuler's included — was built from the same 288 measured patches used for the evaluation. The numbers therefore measure how well a generator fits (interpolates) colours it has already seen, not how well it generalises to unseen colours; there is no hold-out set and no cross-validation. Higher-parameter models benefit most from this setup (LUT-based profiles most of all, low-parameter matrix profiles such as VueScan least).
2. Gamut clipping: when a device RGB falls outside a profile's gamut, LittleCMS maps it onto the gamut boundary, so the resulting ΔE00 contains clipping error rather than interpolation error. The perceptual intent is by specification allowed to compress the whole gamut, so the numbers in 2.3 are not an accuracy metric; of the three intents only absolute colorimetric shares the semantics of the reference data (absolutely measured, paper white L*≈93.8). Relative colorimetric is covered by the last statement.
3. Quantisation uncertainty: the referee chain is 8-bit in / 8-bit Lab out, and Pillow's LAB image quantises a*/b* to integers (step 1.0; L* step 0.39). Measured on this panel, that adds a floor of about +0.26 to the per-patch ΔE00 (≈+0.40 for ΔE76, max 0.70). The floor is of the same order for every participant, so it does not reorder the table but makes the absolute values slightly pessimistic; gaps of the order of 0.1 should not be read into. The project's direct 16-bit CLUT path only covers LUT-based profiles (matrix profiles have no CLUT to read, and v4 mAB tables have to be located heuristically), so this report deliberately uses one real CMM for all nine participants instead.
4. White-point convention: generators do not agree on how to declare the media white point — per spec it should equal the absolute PCS coordinates of the white that the CLUT treats as 100. Declaring the measured paper white (ArgyllCMS, L*≈93.5) versus a normalised white produces a global L* shift under relative colorimetric. That is a convention difference, not a mapping-capability difference: the same profile scores 3.41 there and 0.72 under absolute colorimetric. This is also why absolute colorimetric is the primary metric here.