Practical Measurement and Neural Encoding of Hyperspectral Skin Reflectance

We propose a practical method to measure spectral skin reflectance as well as a spectral BSSRDF model spanning a wide spectral range from 300nm to 1000nm. We employ a practical capture setup consisting of desktop monitors to illuminate human faces in the visible domain to estimate five parameters of spectral chromophore concentrations including melanin, hemoglobin, and $\beta$ carotene concentration, melanin blend-type fraction, and epidermal hemoglobin fraction. The estimated parameters make use of a novel three-stage lookup table search for faster parameter fitting, and drive our skin model for accurate reconstruction of facial skin reflectance response in both the visible domain as well as in the UVA and near-infrared range. We also propose a novel neural network architecture that given our measurements, predicts the five chromophore parameters of our model at the encoder stage and full hyperspectral reflectance response as the output of the decoder stage.


Introduction
Accurate facial appearance modeling has been a topic of extensive research in computer vision and graphics.Several biophysically-based spectral skin reflectance models have been proposed in recent years [1, 7, 10-12, 17, 18].Previous works have either proposed complex biophysical models with a large number of parameters to represent various chromophores in the human skin [7,11,18], or employed simpler models with fewer parameters that can be practically measured [12,17], albeit with reduced model accuracy.These works have mainly focused on the visible domain, in which some facial features that are crucial for skin appearance and protection might be concealed [7].
In this work, we employ a diffusion-based hyperspectral BSSRDF skin appearance model spanning a wide spectral range from 300nm to 1000nm, thus encompassing Ultra Violet (UV), Visible (Vis) and Near InfraRed (NIR).While the employed model makes use of the spectral chromophore parameters proposed in [11], we propose a novel extension of their model to predict skin reflectance response in the UVA and NIR domain (Section 3).Therefore, the employed parameters are melanin concentration C m , dermal hemoglobin concentration C hd , melanin blend-type fraction β m , epidermal hemoglobin fraction C he , and β carotene C bc concentration.The skin model additionally includes two fixed parameters: dermal water fraction C wd and epidermal water fraction C we .The inclusion of β carotene in the model enables more accurate skin appearance simulation, particularly for yellowish and olive skin types.Further, we present a practical measurement approach for acquiring spectral skin reflectance based on our model.Our capture setup consists of a set of desktop LCD monitors for illuminating a face with standard visible spectrum RGB illumination in order to acquire high-quality spectral skin chromophore maps.
Previous works estimate spectral skin parameters to achieve high-quality appearance reconstructions by searching in a LookUp Table (LUT) of pre-computed skin tones the closest RGB values of facial albedo images, on a pixel by pixel basis [12,17].Such an approach is computationally expensive, particularly for models with higher complexity.To address this issue, we propose a novel three-stage LUT search to efficiently estimate the parameters of our model (see Section 4).
An additional issue of the LUT search approach is related to the quantization of the results it produces, due to the sampling resolutions of parameters used to generate the LUT.To overcome this limitation, we design an Encoder-Decoder network to estimate the five spectral parameters at the encoder stage (ChromN N ), and a decoder network (SpectraN N ) for reconstructing the full hyperspectral skin reflectance accurately.Here, hyperspectral reflectance of skin is first compressed using Principal component analysis (PCA), which allows reducing the size of the SpectraN N network, thus speeding up training and testing (Section 6).We validate our model parameters estimated from RGB measurements using comparison of hyperspectral skin appearance reconstructions to reference photographs under various spectral illumination conditions.
In summary, our main contributions are as follows: • We propose a practical hyperspectral skin reflectance model driven by five chromophore parameters spanning a wide spectral range from 300 to 1000nm, and a practical measurement approach for estimating model parameters employing RGB illumination emitted by desktop LCD monitors.
• We introduce a novel three-stage lookup table search for speeding up conventional LUT-based parameter fitting that takes advantage of our spectral measurements.
• We design a neural network architecture to estimate the chromophore parameters of our skin model and reconstruct full hyperspectral reflectance from the input RGB measurements based on a novel PCA analysis that enables reduction of network complexity.

Related Work
Here, we give a brief overview of bio-physical skin appearance modeling, spectral skin appearance measurement, and hyperspectral appearance modeling (including UV and IR) in computer vision, graphics and tissue-optics.
Bio-physical Skin Appearance Modeling: Jensen et al. [16] proposed a practical BSSRDF model to approximate the subsurface scattering for translucent materials with dipole diffusion theory.This work was extended to multipole diffusion theory to account for multi-layered translucent materials, such as human skin [9].Building on these, they introduced a two-layer bio-physical skin model with three chromophore parameters to simulate realistic skin appearance [10].This model was then extended and simplified by Jimenez et al. [17] to make it suitable for practical measurement and prediction of skin color change during facial animation using a four parameter model with only two free parameters used for fitting using a lookup-table (LUT) search.Recently, Gitlina et al. [12] employed the more complete spectral model of [17], with four free parameters to more accurately reconstruct skin appearance using measurements in a multispectral LED sphere.Besides employing LUT searching, they also proposed a cascaded neural network for model fitting and RGB albedo reconstruction.
The spectral skin chromophores modeled in this work are mainly related to melanin and hemoglobin concentrations.However, there are some other blood-borne pigments, such as beta-carotene and bilirubin, which also play important roles in skin color [7,11,18].Krishnaswamy et al. [18] introduced a parametric five-layer skin model called BioSpec to approximate the light interaction within human skin.Donner et al. [11] later proposed a layered heterogeneous reflectance model with lateral inter-scattering of light between skin layers using six parameters and an additional inter-layer absorption to model veins and tattoos in skin [11].Although these models can simulate very realistic skin appearance, high model complexity makes measurements less practical.
Our work follows the line of previous diffusion-based bio-physical skin appearance models.We employ a practical hyperspectral skin model with five free parameters to balance model sophistication and complexity for measurements.We also propose a novel Encoder-Decoder network to predict chromophore maps and reconstruct full spectral skin reflectance given practical RGB measurements.Similar to our work, recent methods have employed neural networks to predict chromophores [12].However, their network architectures focus on reconstructing RGB facial appearance from the estimated chromophores, under a specific illumination spectrum (e.g., D65).Closer to our approach, Aliaga et al. [1] presented a network that directly predicts spectral skin reflectance in the Vis and NIR range.In comparison, our proposed network architecture allows predicting hyperspectral reflectance in a wider range (UV+Vis+NIR), while using only a fraction (∼ 2%) of the overall number of network parameters.
Spectral Skin Appearance Measurement: Jensen et al. [16] proposed a simple technique for measuring RGB optical parameters of materials, including two skin samples, by illuminating the surface of a sample with a tightly focused beam of white light and acquiring a photograph with a color camera to observe the radiant exitance across the entire surface.Tsumura et al. [23] used independent component analysis on photographs to estimate melanin and hemoglobin maps, assuming such chromophores have independent effects on skin color and ignoring the influence of scattering and depth.Donner et al. [11] employed multispectral images of skin patches illuminated by a broadband flash and nine different chosen narrow spectral bands to measure spectral skin reflectance and drive parameter estimation for their model.Jimenez et al. [17] used a non-contact SIAscope T M system [8] with cross-polarized flashes to measure the hemoglobin and melanin concentrations of facial skin while Alotaibi and Smith [2] used a single LED illumination spectrum from photographs.Gotardo et al. [13] proposed a multi-view setup with static illumination for dynamic skin albedo estimation.Park et al. [21] used multiplexed spectral illumination for skin reflectance estimation.Closer to our approach, Gitlina et al. [12] designed a measurement approach using two complementary broad and narrow-band spectral illumination conditions using a multispectral LED sphere.They also proposed a practical skin patch measurement approach using an off-the-shelf dermatological imaging device (Miravex Antera 3D camera).Recently, Aliaga et al. [1] employed RGB albedo measurements of facial skin to estimate chromophore parameters for their Monte Carlo simulation based skin model.Our method takes as input photographs under complementary broad and narrow-band illumination conditions produced by LCD screens to estimate parameters of a diffusion-based hyperspectral skin reflectance model.
Ultraviolet(UV) and Infrared(IR) Modeling: Some deep skin damage gradually changes facial appearance and skin health due to UV light exposure.The prediction of the pigmentation irregularities such as freckles is beneficial for detection of skin aging [6,15] and protect skin from photodamage effects and skin cancer risk [25].In addition to pigmentation, skin hydration is another significant factor to describe skin status [7].Water displays strong absorption in the IR domain, thus IR photographs can provide information about skin hydration [4].
Chen et al. [7] first introduced a hyperspectral skin appearance model to computer graphics in order to comprehensively forward simulate spectral and spatial distributions of light interacting with human skin from the UV to IR domain involving a large set of skin chromophores.While we borrow from this work, we employ a more practical model with fewer parameters to model skin reflectance over a wide spectral range spanning UVA-Vis-NIR domains.

Overview
We propose a diffusion-based hyperspectral skin appearance model driven by five biophysical parameters, namely melanin concentration C m , melanin type blend β m (blend between eumelanin and pheomelanin), dermal hemoglobin concentration C hd , epidermal hemoglobin concentration C he , and β carotene fraction C bc .This work adapts and extends the work of [10,11] into the hyperspectral domain using spectral absorption profiles of the constituent chromophores in the UVA and NIR bands borrowed from [7].Additionally, we add two parameters C wd and C we to represent water in the epidermis and dermis, respectively.The inclusion of β carotene, an important skin chromophore, extends the skin model color gamut, thus improving the simulation of yellowish and olive human skin tones [3,11].Figure 2 shows a small change in the β carotene fraction significantly contributes to the resulting skin color.Figure 3 shows the spectral absorption coefficients of the parameters in our model, spanning from 300nm to 1000nm.The selected model parameters not only determine skin color in the visible domain, but they are also significant absorbers in the UVA range.In the NIR range, while both hemoglobin and β-carotene absorption drop, parameters related to melanin and water still significantly affect skin appearance.Overall, the proposed novel equations of absorption of the epidermis and dermis in the [300 − 1000]nm range are as follows: where λ is the light wavelength in nanometers, C m represents melanin fraction, β m represents melanin type blend between eumelanin and pheomelanin, σ em a and σ pm a are the , with γ being the blood oxygenation ratio between deoxy-and oxy-hemoglobin, fixed as 0.75.C bc and σ bc a respectively represent the fraction and absorption coefficient of β carotene.C we and C wd are water fractions in the epidermis and dermis fixed as 0.2 and 0.6, respectively; the absorption coefficient of water is σ w a .σ base a is the baseline skin tissue absorption.Please refer to the supplemental material for more details of our model and how the LUT data is built.

Acquisition
We employ a practical setup for our measurements using LCD monitors for illuminating a face similar to the recent work of [19].Our setup employs four 4K desktop LCD monitors (Asus ProArt PA279CV) and a set of cameras (Canon EOS M6 Mark II) placed in between the monitors for facial measurements.Unlike [19], the cameras in our setup are cross-polarized with respect to the linearly polarized illumination emitted by the monitors (see Figure 4 (a)).
Figure 4 (b) shows the Spectral Power Distribution (SPD) of the screens white illumination (6500K CCT), as measured by a Sekonic SpectroMaster C700 spectrometer.The peak of this blue illumination is at around 450nm.However, according to Preece and Claridge [22], the optimal illumination to measure melanin is narrowband blue with a peak at around 485nm.Therefore, while directly using the narrow band blue illumination in our setup might not  be optimal for melanin measurement, it provides cues to estimate β carotene, as the maximal absorption of the latter is closely aligned to the peak of blue illumination.In addition, narrowband green illumination can be used to estimate hemoglobin [22].As observed in Gitlina et al. [12], directly observing skin response under narrowband illumination can lead to sub-optimal measurements, due to colors outside the gamut of typical off-the-shelf RGB cameras.Hence, we make use of multiple illumination conditions, such as white illumination, a mixture of white and blue illumination, and a mixture of white and green, and employ a computational scheme based on chromatic adaptation transform proposed by [12] to synthesize sharper blue and green narrowband responses from the mixtures (Figure 4 (b)).Therefore, the dimensionality of the input matches that of the estimated parameters:

Parameter Estimation
The most straightforward approach to parameter estimation is to use pre-computed 5D LookUp Tables (LUTs) of skin colors, one for each of the 3 lighting spectra described in Sec 3.2.For each pixel in the RGB photograph acquired under broadband white light, and in the grayscale synthesized blue and green images, we search the corresponding LUTs, finding the set of skin parameters that minimizes the L2 distance in CIELAB color space (i.e.∆E 76 color difference), thus obtaining the estimated skin parameter maps.Radiometric calibration is achieved by scaling the LUTs intensities by constant scalars, derived by taking photographs of the X-Rite color chart under the same illumination conditions.Additionally, we use the optical flow algorithm on the input images to reduce the impact of subjects' movement during the acquisitions.

Three-stage LUT Searching Method
As shown in previous work, naïve LUT search can provide high-quality results, which we show can be further improved for yellowish and olive skin tones by including β carotene in the model.However, the increased model complexity leads to larger LUTs and longer matching time.
In order to address this issue, we propose a novel threestage matching method given our proposed spectral measurements to take advantage of the larger parameter space, while reducing the computational cost without affecting the quality of the estimated parameter maps.In the following, we provide an overview of the proposed search strategy.
• Compared to the full search, the above strategy allows a significant speed up of the maps estimation, and allows incorporating knowledge of the most suitable lighting condition to estimate a given skin parameter (see Sec. 3.2).In fact, while blue illumination excite melanin and β carotene (stages 1 and 2), green illumination provides useful information about hemoglobin (stage 3).

Estimation Results
Figure 5 (a, b) compares the albedo reconstruction using the three-stage matching method with the ground truth photograph, for subjects with Asia, Caucasian, and South Asia skin types, respectively; (c-g) report the estimated chromophore maps while (h, i) displays the UVA and NIR reconstructions.As it can be seen, for all skin types the reconstructions closely match the photographs.The predicted UV response of Subject 3 is lower than that of lighter skin subjects because higher melanin concentration absorbs UV radiation, which is consistent with [7].
Figure 6 compares the reconstruction of an example us- ing the naive full search matching method (b), which inputs the broadband white along with synthetic blue and synthetic green images, with the reconstruction given by the threestage search (c).As shown in the figure, the results of the two methods are qualitatively similar to the photograph (a).

Skin Appearance Editing
Editing the estimated model parameters allows simulating biophysical changes in skin appearance.By scaling the two melanin-related parameters we can simulate lightening or tanning of skin (C m ), as well as increasing or reducing stubble/facial hair (β m ) present in the input photograph (Figure 7 (a,b)).Similarly, skin appearance can be edited by scaling the beta-carotene parameter C bc (Figure 7 (c)).Editing the two hemoglobin-related parameters C hd and C he allows for changes in facial redness, thus leading to a paler or flushed appearance (Figure 7 (d,e)).Instead of scaling linearly, we apply a simple but effective method to treat two hemoglobin parameters.The scaling equation is ), where i is the index of the value C in the array of C hd or C he with the size of N while s is  the scaling factor.With this scaling, the local variation of hemoglobin distributions is accentuated in the edits.

Ultraviolet (UV) and Near-infrared (NIR) Validation
As previously noted, the chromophores-related parameters used in our model absorb light significantly both in the visible and invisible range.Hence, once the chromophore maps have been estimated using visible light, we can infer skin appearance in the UV and IR domain by plugging in our forward model the corresponding absorption coefficients.
To validate the predicted UVA and NIR appearance, we employ a UV camera (fxo487MXGE) without an IR-cutoff filter to acquire facial images.Subjects are illuminated by five UV LED tubes and an IR LED with emission in the [300 − 400]nm and [700 − 1000]nm range, respectively.
Figure 8 (a-c) compares photographs under visible, UVA and NIR lighting, with corresponding albedo reconstructions by our model which assumes uniform illumination (de).Consistent with the conclusion that UV photos can display more facial details, Figure 8 (b,d) show some conspicuous spots around the cheek and forehead are much more evident in the UVA domain than in the visible domain.
On the other hand, Figure 8 (c,e) displays the photograph and reconstruction in the NIR domain, where the faces look softer and less influenced by pigmentations, and more visible blood vessels.Note that the UVA and NIR photographs are not cross-polarized and contain specular reflection which is not simulated in our reconstructions.Therefore, in Figure 8 (e) we simulate shading due to small frontal area light and render specular reflections.To account for lateral scattering, we apply an image-space scattering approximation method on the reconstructed albedo.Please see details in Supplemental material.

Neural Network
As shown in the previous Section, the use of LUTs to infer model parameters, by searching for the closest match to the color of acquired human skin, can lead to good results.However, even when using the 3-stage approach, parameter estimation is still relatively slow.More importantly, the values contained in the maps are quantized due the discretization of the LUTs.To address these issues, we propose the use of an Encoder-Decoder Neural Network for estimating model parameters and skin reflectance from 300nm to 1000nm, thus covering the UVA + Vis + NIR range.

Network Architecture
Our architecture consists of two networks, as shown in Figure 10.The ChromN N network predicts the five model parameters (C m , β m , C hd , C he , and C bc ) from the 5D multispectral skin albedo input, that is RGB data from acquired photograph under white, synthetic grayscale blue, and synthetic grayscale green illumination.The estimated parameters are fed to the SpectraN N network which provides a neural approximation of our model to output spectral skin reflectance.
However, directly sampling spectral data densely (e.g.[300 − 1000]nm in 2nm steps) as output poses issues of larger network and potentially noisy estimates.Therefore, we apply Principal Component Analysis (PCA) to spectral skin reflectance data generated using our model spanning the entire range of parameters, which leads to a training set of 572,220 spectra.We retain the first 10 principal components, shown in the supplemental material, which overall explains ∼ 100% of the total variance.For more details, please refer to the supplemental material.are Sigmoid and Relu, respectively.All experiments were conducted on Titan X GPUs.The loss function for both networks is given by the Mean Squared Error(MSE).  to LUT search.Note that the comparison is done to LUT search for the UVA and NIR cases in the absence of reference photographs.

Quantitative Analysis
We report a quantitative analysis of our methods and the conventional LUT search method towards efficiency and accuracy.The results we show below are obtained by testing all the datasets including different skin types and genders.
-Efficiency analysis: For 2K images, the conventional LUT search method requires 2.88h while the proposed three-stage LUT search uses 13.91min and the PCA network only needs 2.27s on average.
-Accuracy analysis: Table 1 compares MSE loss for reconstructing spectral reflectance over the Vis.domain.This table validates our conclusion that the quality of the threestage method is close to conventional LUT search, while the network addresses the discrete value problems of LUTs and achieves much higher accuracy.Our method is not too biased toward certain skin tones.

Correspondence Comparison
To validate our model, we compare the generated skin reflectance with the accurately measured data in Leeds skin dataset, which is limited to the Vis domain [24] (Figure 11); complete hyperspectral skin reflectance from our model is shown in the supplemental.The maximum and average RMSE between our model and all the data in the Leeds dataset is 0.036 and 0.008, respectively, which is lower than that of Aliaga et al. [1] (mostly lower than 0.05).Figure 12 illustrates a UV photograph alongside reconstructions using multiple inputs (white, synthetic blue, and synthetic green) and a single input (white) in the UVA domain.While reconstructions in the visible domain using both sets of inputs closely match the photograph, significant differences emerge in the UVA domain.Specifically, the reconstruction with multiple inputs (b) more closely resembles the original photograph (a) compared to the single input (c), benefiting from narrowband measurements.
In contrast to Aliaga et al. [1], who use a Monte Carlo simulation-based model with a single input, we employ a diffusion-based skin model utilizing multiple inputs.Our model demonstrates higher accuracy and better performance across a broader domain, and has been validated by practical measurement.Furthermore, our PCA-based network has ∼ 2% of the parameters compared to the network used by [1], indicating a more efficient approach.Please see more details in the supplemental material.We also benchmarked our method against Gitlina et al. [12], using their Light Stage data.Figure 13

Conclusion
We proposed a practical diffusion-based hyperspectral skin reflectance model driven by five chromophore parameters which faithfully represent spectral reflectance from 300 to 1000nm range.To estimate such parameters for faces, we presented a practical measurement setup that employs desktop LCD monitors for spectral illumination.For efficient parameter estimation, we introduced a new threestage LUT-search method.Moreover, we propose a novel compact neural network to estimate the parameters and reconstruct full hyperspectral reflectance using PCA-bases.
Our practical measurements and model allow accurate prediction of skin appearance under a wide range of spectral illumination conditions, as well as realistic biophysical editing of facial appearance for various applications.
In this section, we provide details on our model and how the LUT data is generated.
-Diffusion BSSRDF: Similar to the work [10,12], we used a two-layer skin model, which consists in the epidermal dermal layers, based on diffusion theory.For the epidermis, we employ a multipole diffusion model to simulate subsurface scattering, with thickness fixed to 0.32mm.As for the dermis, simulated as a semi-infinite medium, we employ a dipole diffusion model.The absorption coefficients for epidermis and dermis are shown in Equation 1 and Equation 2.
The reduced scattering coefficient of the epidermis is The scattering coefficient of the dermis is approximated as half of epidermal scattering [10] [5].
The index of refraction (IOR) of both layers is 1.4.We convolved the spectral profiles (reflectance and transmittance) with the Kubelka-Munk theory [14] to obtain the total diffuse reflectance, then generated a LUT with SPDs of different illuminations.
-Chromophore sampling: For each point in the LUT, we sampled five chromophore parameters in different ranges and methods as shown in Table 1, parameter range and sampling density vary on a per-chromophore basis.C m , C hd and C bc are sampled cubicly in their respective ranges, according to the following equation: where c i is the value of the i th sample for chromophore C, i ∈ {0,

B. Narrowband measurements
As observed in previous work [12], narrowband illumination allows to reduce the baked-in subsurface scattering in the estimated chromophore maps and hence in the albedo, thus leading to sharper details than the corresponding estimates from broadband-only measurements.The increased sharpness is beneficial when rendering skin appearance, as the blurring introduced by subsurface scattering simulation allows a closer match to the ground truth appearance.On the other hand, estimates obtained under broadband illumination will appeared over-blurred when rendered.However, as discussed in the main paper, to address camera color gamut issues we avoid direct observation of skin response under narrowband illumation and use a computational scheme to recover the response from the mixture narrowband + broadband.Figure 1 compares the reconstruction of cheek area using broadband white and synthetic blue and green illuminations (a) with the ground truth photograph (b).In the same figure, we report the reconstruction using broadband white and direct blue and green illuminations.As the figure shows, the reconstruction with synthetic data shows sharper skin details than that reconstructed from measurements direct illumination, particularly visible around skin blemishes (e.g.moles) and pores.

C. Image-space Scattering Approximation
In our work, both the proposed LUT search or the neural network aim at producing sharp reconstructions, thus reducing the effect of lateral scattering of light.As observed in the previous section, this would affect the realism of skin appearance, if the reconstructed albedoes were directly compared with ground truth photographs.To address this, we apply an image-space scattering approximation method on the reconstructed albedo which produces images that better match the validation photographs.We first select the closest skin reflectance curves in our lookup table to those of the light and dark skin patches in a standard colorcheker chart.For each patch, we select the reflectance values at 450nm, 530nm, 630nm and 835nm, corresponding to the peaks of blue, green, red, and NIR illumination in our model, respectively.Similar to the work of [20], we apply a linear combination of seven Gaussians to fit the curves of each patch.Note that these Gaussians share the same variances, and each profile weights them differently shown in Figure 2. We then blur the reconstructed albedos with each Gaussian and obtain the final result via a linear combination of multiple blur kernels.We do not apply any blur to the UVA reconstruction since this has been predicted from measurements under longer wavelengths (i.e.visible light), thus already encoding some blur.
Figure 3 compares the results and photos under white (a), red (b), green (c), blue (d), and IR (e) illuminations.As for the broadband white image, we process RGB channels separately, since the white illumination of the monitor setup is simply a sum of three separate RGB peaks.Finally, for the IR LED panel, we simulate shading due to small frontal area light and also render specular reflection.We brighten these images to clearly show the details.As the figure shows, adding the image-space blur makes the results qualitatively closer to the photographs.

D. Neural Networks
In this part, we offer detailed information of the PCA-based network, ablation study of PCA, and comparisons with the recent work of Aliaga et al. [1] We apply Principal Component Analysis (PCA) on the full spectral skin reflectance data spanning the UVA, Vis, and NIR domains to reduce the complexity of the proposed neural network (see Section 6).The retained first 10 principal components, which overall explain ∼ 100% of the total variance, are shown in Figure 5.Using such components, any given skin reflectance spectrum can be reconstructed using a 10d vector containing one weight for each of the 10 principal components, along with the mean spectral reflectance of the dataset.
-Ablation study: Figure 4 compares a ground truth skin reflectance spectra with its PCA reconstruction using this 10d PCA basis.As can be seen, the reconstruction almost perfectly matches the spectral reflectance, sampled at 2nm resolution.
Compared with the recent work of Aliaga et al. [1], which directly estimates reflectance without PCA, our method offers three advantages: • Our model and PCA-based neural network covers a wider spectral domain spanning from UVA, visible and NIR domains, while Aliaga et al. [1] only predicts reflectance in the visible and NIR domain.• The predicted skin reflectance using our model is more accurate.Figure 11 compares the generated spectral skin reflectance with Leeds skin dataset [24] of two examples representing light and dark skin.As can be seen, the generated skin reflectance of our model is close to that measured accurately in the Leeds dataset.As mentioned in Sec 7.2, the maximum and average RMSE of our model is 0.036 and 0.008, respectively while the RMSE between Aliaga et al. is generally lower than 0.05 [1].The accuracy of the model will decide the performance of the network since the training dataset comes from the LUTs generated by our model.• Although our method can cover a wider spectral range and at higher resolution, the size of our network is smaller, with significantly fewer parameters (9354 parameters for our network, thus about 98% fewer parameters than the 424758 for the network in [1]).
Figure 5.The first ten PCA components derived from our analysis.

Figure 2 .
Figure 2. Spectral skin reflectance variations due to C bc .Cm increases along the x-axis, C hd along the y-axis; βm and C he are kept fixed to 0.5 and 0.25, respectively.

Figure 3 .
Figure 3.The spectral absorption coefficients curves for the skin chromophores included in our model from 300nm to 1000nm.

Figure 4 .
Figure 4. Monitor-based setup and the SPD of white illumination.
Figure 5.Comparison of photos (a) and reconstructions (b) and estimated spectral parameters (c -g) for faces of Subject 1 (Asia skin type), Subject 2 (Caucassian skin type), and Subject 3 (South Asia skin type).The estimated chromophore parameters also predict skin appearance in the UVA and NIR bands (h, i).

Figure 6 .
Figure 6.Comparison of photograph (a), reconstruction using the naive LUT search method, using white, Synth.blue and Synth.green (b), and reconstruction using the 3-stage search (c).

Figure 8 .
Figure 8. Comparisons of Vis, UV and IR photos (a-c) and reconstructions (d, e) using our proposed method.

Figure 9 .
Figure 9.Comparison of the network predictions and references of subject 1 (Mediterranean skin type) and subject 2 (Asian skin type).Figure(a-e) compare the chromophore estimation between the ChromN N outputs and LUT searching methods with the same inputs.Figure(f-i) compare references and visualizations of facial reflectance produced by SpectraN N in the visible domain under white (f), red (g), green (h), and blue (i) illuminations, and in the UVA (j) and NIR (k) domains.Here, the references for the visible domain are photographs, while the references for the UVA and NIR domains are the reconstructions generated by the proposed three-stage LUT searching method.

Figure 9 (
Figure9(a-e) compares the outputs of the ChromN N network and the chromophore maps generated by the proposed three-stage LUT search method of two subjects (Mediterranean and Asia skin types).It is worth noting that the chromophores maps obtained from LUT search are not ground truth, but they can be used as a reference.Generally, the network results have similar quality to the reference maps from LUT search.Chromophore maps from the neural architecture show less noise and less quantization than the ones from LUT search, particularly noticeable in the β m maps and C bc maps.In order to display clearly, we scaled all chromophore maps, especially C bc maps which should have little difference shown in the range from 0 to 0.01.After feeding the outputs of ChromN N into the SpectraN N , we can obtain the estimated 10-value PCAbased skin reflectance, then expanded into full spectral reflectance.Figure9 (f-i) compares photographs and corresponding network-generated reconstructions, under white (a), red (b), green (c), and blue (d) visible spectrum illumination produced by the LCD screens.Additionally, in Figure 9 (j,k) we compare network-reconstructed facial appearance in the UVA and NIR domains with the corresponding values provided by LUT search.As the figure shows, the network prediction results are qualitatively very similar

Figure 10 .
Figure 10.The proposed neural architecture consists of two networks.The ChromN N maps multispectral albedo data to the five spectral model parameters.The model parameters are then fed to the SpectraN N to predict a PCA-based, low-dimensional skin reflectance representation, finally reconstructed using the standard PCA reconstruction technique to obtain the full spectral reflectance in the UVA+VIS+NIR range (300nm to 1000nm).

Figure 11 .
Figure 11.Comparison between the generated skin reflectance of our model and Leeds skin dataset of dark and light skin.
displays the photograph of a subject under white illumination (b), its 4D model reconstruction by Gitlina et al. (a), and our 5D model reconstruction (c), that incorporates the parameter C bc , with fixed C we and C wd .The absolute error maps displayed in the right corner insets show that our model provides more accurate reconstructions than Gitlina et al.'s, and that it is not limited to our data acquisition setup.

Figure 1 .
Figure 1.Comparison of photograph and reconstructions with different measurements.Figure (a) employs broadband white, synthetic blue, and synthetic green illuminations while Figure (c) employs broadband white, direct blue, and direct green illuminations.

Figure 4 .
Figure 4. Comparison be a ground truth skin reflectance spectra with its PCA reconstruction using the first 10 PCA components.

Table 1 .
Accuracy analysis

Table 1 .
1, . . ., N }, N + 1 is the number of samples and C max is the maximum value C can assume in our model.C m , C hd and C bc have 51, 51, 5 samples, respectively.β m and C he are uniformly sampled, respectively with 11 and 4 samples.Parameters of skin spectral absorption and scattering.