Support for Kubelka-Munk theory "pigment-based" color mixing

[bj-rn]

Similar to mixbox (commercial).

• "Practical Pigment Mixing for Digital Painting" research paper
• mixbox code on github
• MIT implementation of Kubelka-Munk

[waacton]

There are a lot of moving parts to be considered!

Similar to CMYK where ICC profiles are not restricted to 4 colour prints (e.g. CMYK vs CMYKOGV), I'll need to make some decisions around handling N-channel spaces. The paper uses 4 primary pigments, whereas spectral.js appears to use 7.

The paper recommends defaults (Phthalo Blue, Quinacridone Magenta, Hansa Yellow, Titanium White) but ultimately the pigments used are the artist's choice, so I'd want to support customisation. spectral.js uses hardcoded SPDs and I don't know what pigments they refer to. I'm also not familiar enough to know why only SPDs are being used in spectral.js and not absorption and scattering coefficients as described in the paper.

Making things more difficult, it seems like the kind of datasets used in the paper are unavailable, unless I'm missing something?

• Artist Paint Spectral Database - the link to "download Excel spreadsheet" just redirects
• Artist Acrylic Paint Spectral, Colorimetric, and Image Dataset - has a note saying "the spectral reflectance database mentioned in the article is no longer available"

I will probably investigate a first pass at a not-fully-integrated version (as with CMYK), if I can find suitable data.

[tebjan]

Back reference to where the idea started: https://discourse.vvvv.org/t/vl-mixbox/22649

[bj-rn]

Hey, thanks a lot for looking into this! TBH I didn't really read the paper...

Concerning your questions regarding spectral.js maybe this issue provides some answers.

I was able to find the Excel spreadsheet using wayback machine:
https://web.archive.org/web/20190715095836/https://www.rit.edu/cos/colorscience/mellon/Data/Final_artist_database.xlsx

[waacton]

Ah excellent, that makes a bit more sense now, I'll take a look when I get a chance!

There's also more detail about the spectral.js implementation here but I'll need more coffee before I try to follow along 😄

[waacton]

One more thing while it's on my mind: if this does lead to some notion of Pigment in Unicolour, it'd be really convenient if it could be somehow generated from data found on https://infraart.inoe.ro/ (if it's even possible to download the raw data).

[waacton]

I've not forgotten about this, but I wanted to figure out how to support genuine CMYK using ICC profiles first as it's a much more common use case.

There's still work to do there (notably for ICC v4 profiles), but it made me wonder: could "pigment-based" mixing (at least for the suggested defaults Phthalo Blue, Quinacridone Magenta, Hansa Yellow, Titanium White in the first instance) be captured as an ICC lookup transform (4-colour to LAB D50 and reverse)?

I don't know anything about creating an ICC profile but it might be a shortcut to supporting a first pass without customisation of pigments. The ICC have a registry of CMYK profiles that each has its characterisation data reference e.g. FOGRA39. Perhaps with similar data for pigment-based mixing, the conversion could be distilled down to an elaborate series of lookup tables for convenience🤔?

[bj-rn]

I haven't a clue, tbh.

[waacton]

I've spent some time looking into this on the pigments branch and wanted to record my thoughts while they're fresh in my mind.

I'm confident I can support some level of Kubelka-Munk mixing but I'm not yet feeling comfortable about it, and I'll need to ruminate and experiment some more.

I've got a notion of a Pigment which is created using absorption (k) and scattering (s) spectral data

• this is the data format provided in the (no longer available) Artist Paint Spectral Database and used by the Mixbox paper Practical Pigment Mixing for Digital Painting
• I think this is the two-constant Kubelka-Munk theory
• k and s are used to calculate reflectance at each wavelength
• reflectance is corrected using the Saunderson correction, though a variant that assumes spectrophotomeric measurements were made in SPEX mode (specular excluded)
  • I don't know if this is a safe assumption to make (potentially since it mimics how eyes perceive colour), or if I'd need to expose SPEX mode as a parameter
• reflectance is used alongside spectral power distribution to calculate an XYZ colour
• however, k and s are likely to be a different range and interval of wavelengths
  • right now I'm simply linearly interpolating for missing values
  • again, I don't know if that's a reasonable assumption to make
• with all these caveats in place, I can show pigments mixed in a similar way to the images in the Mixbox paper (see below)
  • it's not a perfect match but I don't know the details of their implementation of K-M, or what processing has been done on the images themselves
• however, the colours created from pigments are quite different from the LAB values reported by the paint manufacturer
  • no idea if I should be expecting them to, or whether the above methodology is simply too different to the manufacturer's

In the following images:

1. Left is the K-M demo image from the Mixbox paper
2. Middle is the real paint mixing image from the Mixbox paper
3. Right is my own implementation of K-M, calculated using k and s from Artist Paint Spectral Database (with Saunderson correction constants also from the database, assuming SPEX mode)

Quinacridone Magenta to Titanium White	Phthalo Blue RS to Titanium White	Cobalt Blue to Hansa Yellow Opaque

I could also/instead have a notion of a Pigment which is created using absorption-to-scattering (k/s) ratios

• this is the data format provided in HB 10 mil Drawdowns over White, apparently directly from the manufacturer and found on here
• I think this is the single-constant Kubelka-Munk theory
• k/s is used to calculate reflectance at each wavelength, which can also be corrected if constants are provided, and are again unlikely to be the same wavelength set as the SPD so interpolation may be needed
• The results look very different, but the colours created from these pigments match the manufacturer's LAB values very well
• Sadly there is no Titanium White or Hansa Yellow Opaque data to make a comparison with the reference images from the Mixbox two-constant approach

So the big questions for me to consider:
1. Does a Pigment accept two-constant k and s data, single-constant k/s data, or both?
2. Does Saunderson correction always assume SPEX mode, or is that yet another variable to consider?
3. Is my understanding of all the jargon and terminology even correct?
4. Will any user realistically have any of this data to provide?
5. Is a small dataset of predefined pigments (and Saunderson correction constants) enough to make this feature more approachable?

Finally, Mixbox goes a step further which is out of scope for Unicolour: "unmixing" RGB colours back into some approximation of a pigment for the interactive painting. As far as I understand this is done by precomputing a model based on 4 pigments, and a different choice of pigments would require recomputing the model. I don't see this as feasible at runtime for a library.

[waacton]

Finally, Mixbox goes a step further which is out of scope for Unicolour: "unmixing" RGB colours back into some approximation of a pigment for the interactive painting. As far as I understand this is done by precomputing a model based on 4 pigments, and a different choice of pigments would require recomputing the model. I don't see this as feasible at runtime for a library.

Before I shut the door on this, I'll investigate generating reflectance curves from RGB through this method: http://scottburns.us/reflectance-curves-from-srgb-10/.

(Given there's no "true" answer and this is finding one of infinite possibilities that would generate a target RGB, any implementation would probably end up as some kind of utility outside of the core Unicolour functions. To let people imagine "but what if I had a pigment that actually looked like #FF1493?")

If possible, this would necessitate at least the single-constant implementation (a reflectance value per wavelength, not a k and an s per wavelength). But if the Mixbox-style two-constant implementation seems more realistic and desirable, I might well try to support both, even if the two-constant variant is harder to obtain data for and much less likely to be used outside of a predefined dataset that Unicolour provides.

I'll see where this leads anyway.

[waacton]

What's been bothering me about both Mixbox and Spectral.js is that they start with concrete Kubelka-Munk theory before extending into something less tangible.

The goal of Unicolour is to be accurate, but it can't be accurate when there isn't a correct answer. I want Unicolour to be a library of certainties, and both of these implementations introduce uncertainties.

Mixbox ¹

• The more compelling of the two implementations, every step seems to have a clear purpose
• Uses two-constant KM to give more realistic results (note the dark violet of Phthalo Blue transitioning through blue)
  
• Configuration is needed: the knowledge and selection of 4 pigments' absorption and scattering coefficients
  • A faithful reproduction of Mixbox could just use their choice of Phthalo Blue, Quinacridone Magenta, Hansa Yellow, Titanium White
• Requires solving of an optimisation problem to determine pigment concentrations, either at runtime (which I would choose to do to allow configuration) or beforehand to create a less precise all-purpose model (which I think Mixbox implementation does)
  • This is the bit I don't like, leaves the end result open to interpretation / implementation decisions
• Requires solving of an optimisation problem to map chosen pigments to ones that result in in-gamut RGB values when mixed
  • I have to admit, at this point I started losing interest. Any non-predefined model would involve multiple solvings.

Regardless of how compelling it is, the code they provide is licensed under the somewhat restrictive Creative Commons Attribution-NonCommercial 4.0. If I ever want to provide a Mixbox-style implementation, it will be entirely my own work based only on the paper, and even then I'm not entirely sure that's enough.

So for that reason I focused my attention on...

Spectral.js ²

• Looks logical on the surface with claims like "designed to deliver realistic color mixing" and "mimics real-world color interactions", but the more I dig the more I question whether that's true, and there's no comparison to mixed real-world colours to back it up (unlike the Mixbox paper)
  • The author even says in an issue "this library is not made to be scientifically accurate, it's made for aesthetically pleasing colors"
  • One particular aspect seems to have no rationale behind it other than it results in something that looks nice
• Uses single-constant KM for more naive results (note the dark violet of Phthalo Blue transitioning only through violet)
  
• Fundamentally requires being able to generate reflectance R from RGB colours, either to predefine some reflectance curves to use as a way of approximating a given RGB colour (Spectral.js implementation), or to just generate reflectance for any RGB colour at runtime (which I would choose as the more accurate option)
• Requires solving of an optimisation problem to find a reflectance that results in a specific RGB
  • Again, I don't particularly like this for Unicolour as there are infinite possible reflectance curves for a single RGB, what makes the one I choose the right one?
• The real secret sauce here is that it performs KM as normal, just with a modification to the requested interpolation distance based on luminance, for artistic reasons

Unicolour

All this said, I've ended up with some prototypes that produce convincing results. For a quick comparison:

	Cobalt-Blue-to-Hansa-Yellow	Phthalo-Blue-to-Titanium-White
Reference
Unicolour (2 constant data)
Unicolour (1 constant data)
Unicolour (1 constant data + Spectral.js weighting)
Spectral.js

The least convincing is the standard single-constant KM implementation, but I've got no reason to believe it's wrong, especially when simply tweaking the desired pigment concentrations gives very similar results to Spectral.js.

Additionally, now that I see what's going on with Spectral.js, it's easy enough to extend it to support mixing of N colours, not limited to 2 (and not currently implemented in Spectral.js). Here are the above gradients with a constant small amount of a third pigment mixed in as well:

	Cobalt-Blue-to-Hansa-Yellow-with-Cadmium-Red	Phthalo-Blue-to-Titanium-White-with-Phthalo-Green
Unicolour (2 constant data)
Unicolour (1 constant data)
Unicolour (1 constant data + Spectral.js weighting)

Plan

I think there is merit in incorporating things that are useful or interesting, even if they come with a certain level of ambiguity, with no exact "correct answer". But I don't feel like they belong in the core Unicolour library, which is all about correct answers. So my intention is to make an additional package - Unicolour.Experimental - that enables things like this.

1. Add Pigment to Unicolour, capable of performing core KM calculations (both two-constant and single-constant) as long as user has pigment measurements
2. Add a dataset of two-constant pigments to Unicolour.Datasets to allow easy mixing of those specific colours
3. Add the ability to generate a reflectance curve from an RGB colour to Unicolour.Experimental, effectively enabling any user to approximate a single-constant pigment of any RGB colour
4. Add an implementation of Spectral.js to Unicolour.Experimental, but extended to support mixing > 2 colours

I expect this will take at least a few weeks or months. Items 1 & 2 are largely complete in the pigments branch but I need to find ways to test. Item 3 is the trickiest, involves some hefty matrix wrangling and some LU decomposition that is all extremely rough right now. But since item 4 is just preprocessing concentration parameters before mixing, it's basically done as soon as item 3 is done.

If all goes well, I might even look into a custom MIT implementation of the Mixbox algorithm one day, for more realistic two-constant pigment-style mixing of any RGB colour.

Footnotes

1. Mixbox summary:

   1. Know absorption k and scattering s of 4 pigments
      1. Solve an optimisation problem to find 4 pigments whose mixtures k and s are entirely within RGB gamut
      2. Use these in-gamut pigments instead
   2. Take an RGB colour
   3. Solve an optimisation problem to find a concentration of the 4 pigments that results in the closest RGB colour
   4. Note how far from the target RGB it is (the residual)
   5. Linearly combine the concentration and the residual with other colours
   ↩

2. Spectral.js summary:

   1. Know reflectance R for sRGB red, green, blue, cyan, magenta, yellow, white
   2. Take an RGB colour
   3. Generate a reflectance R for this colour using linear RGB to decide how much of each predefined Rs to apply
   4. Linearly combine the reflectance R with that of another colour, using the requested interpolation distance
   5. But first modify the requested distance by some formula for which no reference has been provided
   ↩

[bj-rn]

Not sure if these will help in any way but thought I'll leave them here just for reference:

• Pigment Painter GPL implementation of the Mixbox paper.
• Feature request for adding MixBox to Krita
• PR adding spectral blending to Krita
• Adding "real color blending" to MyPaint