Fusion of fNIRS and EEG Signals: Mental Stress Study

— Fusion of Functional Near infrared Spectroscopy (fNIRS) and Electroencephalograph (EEG) is a novel approach. This study aims in improving the detection rate of mental stress using the complementary nature of fNIRS and EEG neuroimaging modality. Simultaneous measurements of fNIRS and EEG signals were conducted on 12 subjects while solving arithmetic problems under two different conditions (control and stress). The stress in this work was based on arithmetic task difficulty, time pressure and negative feedback of individual performance. The study demonstrated significant reduction in the concentration of oxygenated hemoglobin (p=0.0032) and alpha rhythm power (p=0.0213) on the PFC under stress condition. Specifically, the right PFC and dorsolateral PFC were highly sensitive to mental stress. Using support vector machine (SVM), the mean detection rate of mental stress was calculated as 91%, 95% and 98% using fNIRS, EEG and fusion of fNIRS and EEG signals respectively.


I. INTRODUCTION
Functional Near-infrared Spectroscopy (fNIRS) is a noninvasive brain imaging technology based on hemodynamic responses to cortical activation.fNIRS measures blood flow through hemoglobin concentrations and tissue oxygenation in the brain [1].It sends near-infrared light into the head using two wavelengths of 695nm and 850nm.By measuring the light sent at two wavelengths, the change in oxygenated 2

O Hb and deoxygenated
HHb hemoglobin concentrations can be estimated using modified Beer-Lambert law [2].fNIRS has several advantages compared to other neuroimaging modalities.Compared to functional magnetic resonance imaging (fMRI), fNIRS has good temporal resolution, portable, inexpensive and less motion artefacts [3].Furthermore, fNIRS has a good spatial resolution compared to EEG.
In recent years, fNIRS has attracted considerable attention as a non-invasive neuroimaging technique for the assessment of hemodynamic alterations in the brain [4].Various cognitive tasks were performed to understand the relationship between the hemodynamic response of the prefrontal cortex and different events.Mental rotation [5], word generation [6], listening to music [7], and arithmetic task [8] have shown to create a hemodynamic response in the prefrontal cortex (PFC).
The change in concentration of the 2 O Hb and HHb hemoglobin has been used in previous studies to classify the brain activation from that of rest state.Study in [7] used a piece of music as stimuli to discriminate between the brain activation and baseline resulted in an average classification accuracy of 70%.Another studies [9,10]used music and arithmetic tasks to investigate the consistency of single-trial classification over multiple sessions resulted in accuracy of 62.7% and 71.2% in classifying music and arithmetic tasks respectively.A comparative study on the classification of three mental tasks was firstly conducted by [11].
The study reported an averaged classification accuracy of 71% for mental arithmetic, 70% for word generation and 62% for mental rotation.However, the hemodynamic activation is an intrinsically slow response.To overcome this limitation, combining multiple neuroimaging modalities with complementing strength is the main objective.Electroencephalogram (EEG) has an excellent temporal resolution enabling it to measure cognitive changes within millisecond scale.EEG is one of the most common sources of information used to study brain function and condition and can be recorded non-invasively using surface electrodes on the scalp.Combination of fNIRS and EEG has recently introduced.Few studies have used EEG and fNIRS to study the correlation between cortical activation and the hemodynamic response in human subjects [12,13].These studies found a positive correlation in occipital cortex between alpha activity and concentration changes of HHb hemoglobin.Another studies combined EEG and fNIRS signals to characterize the hemodynamic response to epileptic discharges as measured by EEG [14].Simultaneous measurement of EEG and fNIRS has been proposed for improving the performance of brain computer interface systems (BCI) [15].Fazly et al. reported that, the performance of a sensory motor rhythm (SMR) based BCI significantly improved by simultaneous measurement of EEG and fNIRS [16].In this work we examined the possibility of combining the hemodynamic responses to mental stress with their physiological counterparts in a multimodal fusion technique.We aim to improve the detection rate of mental stress using the complementary nature of fNIRS and EEG neuroimaging modalities.

A. Subjects
Twelve healthy male right-handed adults participated in the simultaneous EEG and fNIRS measurements.All participants were informed prior to the experiment and gave written consent, in accordance with the declaration of Helsinki and ethical approval granted by local ethical committee.None of these participants had a history of psychiatric, neurological illness or psychotropic drug use.The participants were asked to minimize their head movements and to keep calm as much as possible during the experiment.

B. Experiment setup
To measure the hemodynamic response we used an OT-R40 (Hitachi Medical, Japan) Optical Topography system.Brain activities recorded from the PFC cortex using integrated cap of BrainMaster (7-Electrodes) and fNIRS (27 channels) as shown in figure 1.The sampling rate for EEG recording was set to 256 Hz and 10 Hz for fNIRS.The impedance of EEG was minimized using small amount of gel directly to the scalp.The mental stress experiment was designed based on Montreal Imaging Stress Task (MIST) [17].The task involved three single-digit integers (ranging from 0 to 9) and the operands limited to '+' and '-' (example 4-2+2).The experiment was conducted in four phases.First, brief introduction was given to the participants.Second, all participants were trained for five minutes to estimate the average time taken to answer each question.Third (i.e. the control phase), the participants had their simultaneous EEG and fNIRS data record for total of five minutes.During the control phase, all participants were instructed to answer the questions as fast as they can.Fourth, (i.e.stress phase), the averaged time recorded during the training phase was reduced by 10% as time limit set for answering questions in the stress phase to induce stress on the participants.Furthermore, the individual negative feedback of answering the questions was displayed on the computer monitor to further induce more stress on the participants.The designed experiment consists of four blocks as shown in figure 2. In each block, mental arithmetic task posed for 40 seconds followed by 30 seconds rest.In this experiment, we developed the control technique using MATLAB to send a marker via parallel port of channels 23-24 of EEG brainMaster as '1' to mark the start of the task and '0' for the end of the task for each block.The same marker was sent through serial port to mark the task in fNIRS system as 'F9' for starting the task and 'F7' to mark the end of the task in each of the experimental block designed.The overall time taken for the entire experiment was less than 18 minutes.

C. Measurement Locations
We investigated the EEG at seven electrode locations namely a: FP1, F3, F7, Fz, FP2, F4 and F8 with one reference electrode A1 attached to the earlobe.These locations were based on the 10-20 international system of electrode placement.A total of 27 locations over the frontal lobe were initially examined with fNIRS to determine the extent to which 2 O Hb concentration in the PFC was influenced by the mathematical task.Seven of these locations were subsequently formed the basis of the fusion in this study.

D. fNIRS data analysis
The data collected by fNIRS passed through several preprocessing steps to filter the high frequency components and remove the motion artefacts using the plug-in analysis software Platform for Optical Topography Analysis Tool (developed by Hitachi, Japan; run on MATLAB).The signals were bandpass filtered from 0.0125 to 0.8 Hz using 4 th order Butterworth filter.In baseline correction, we defined a period from 5s prior to task condition to the end of the period of rest condition as the analysis experiment.Then we applied linear regression by least mean square method during the 5s period and the last 1s period to determine the linear trend of the baseline unrelated to the arithmetic task [18].After correcting the baseline by removing the trend, we averaged the baseline corrected data to all the analysis blocks [19].From the analysis blocks, we extracted the average of the change on oxygenated hemoglobin 2 O Hb using the following equation:

E. EEG Analysis
The raw data from EEG was bandpass filtered between 1 Hz and 30 Hz using 3 rd order Butterworth filter.The artefacts were removed using independent components analysis technique (ICA) available in EEGLAB.Then we used wavelet transform to decompose the signal into four frequency bands namely; Delta (0-4Hz), Theta (4-8Hz), alpha (8-16Hz), and Beta  in the same manner of our previous study [20].In this study, only alpha frequency band was considered for features extraction.We selected alpha rhythm due to its significant responses to mental stress.From the wavelet coefficients we extracted the average power of alpha frequency band using the following equation: where x(n) represent the segmented EEG signal and N is the length of the signal.

F. Fusion of fNIRS and EEG signals
The fusion of both modality was performed based on concatenating the features of both modality into a single feature vector.In this study, seven fNIRS channels were fused with seven EEG electrodes.The selection of channels was based on close proximity to EEG electrodes following the 10-20 system.For example, the features of channel 26 were fused with that of FP1, features of channel 24 were fused with that of FP2, features of channel 23 were fused with that of F8, features of channel 27 were fused with that of F, features of channel 1 were fused with that of F4, features of channel 3 were fused with that of Fz and features of channel 5 were fused with that of F3.All features were concatenated to form a single feature vector which later go for classification.In short, the fusion here was based on spatial location of Electrodes placement.

G. Classification
Signal classification was performed on the fNIRS and EEG signals following the processing steps described previously.We aim to classify the activity into one of two classes; 'stress' and 'control'.The classification was done in two phases (individually and after fusion).First, fNIRS and EEG modality classified separately.Second, the simultaneous bi-modal (fNIRS-EEG) was classified after fusing their features and their performance was compared to individual modality.In this work, we used support vector machine (SVM) classifier and calculated the classification accuracy via leave-one-out cross-validation.In particular, for a number of trials (N), N-1 trials were used for training and the remaining 1 trial used for testing.

A. fNIRS
With regard to blood flow in the PFC, the level of oxygenated haemoglobin dramatically increased during a control task (compared to baseline) and significantly reduced during a stress task (compared to control condition) in all the subjects.Figure 3 and figure 4 show the topographical map of oxygenated haemoglobin concentration during the control task and stress task respectively for averaged of 12 subjects.From figure 4, the reduction of oxygenated haemoglobin concentration was much localized to the right PFC and dorsolateral PFC.This indicated that, the right PFC and dorsolateral PFC are the most sensitive brain region to the detrimental effects of mental stress.The study concluded that, with increasing time pressure (i.e.stress condition), the blood flow in the prefrontal area and the information processing abilities significantly decreased.Using two sample t test, the significant differences between the control task and stress task was measured with average pvalue of 0.0032.

B. EEG
EEG results demonstrated significant decreased in alpha rhythm power under stress task in all the recorded electrodes positions.The decreased in EEG alpha rhythm power reflect the increase of stress level on the PFC. Figure 5 shows the mean alpha power distribution across all the recorded electrodes under the control and stress task for average of 12 subjects.Using two sample t test, the statistical analysis demonstrated significant reduction in alpha rhythm power under stress task with mean p-value of 0.0213.The classification accuracy obtained from fNIRS signals was measured as 91% and the classification accuracy obtained from EEG signals was measured as 95% as shown in figure 6.However, the results of the classification of both modalities reported significant improvement in the detection rate of mental stress as compared to single modality.Under fusion of both modalities, the study reported 98% classification accuracy indicating that fNIRS and EEG modality complement each other in obtaining features that highly correlated with mental stress.This improvement is in line with our previous stress studies [21][22][23][24][25][26][27][28][29][30][31][32].

IV. CONCLUSION
This study investigated the effectiveness of simultaneous measure of EEG and fNIRS in studying the effects of mental stress to PFC area.The results showed that, the oxygenated haemoglobin concentration and the alpha rhythm power significant reduced under stress condition.Furthermore, the study reported that, combination of both modalities improve the performance of mental stress detection rate by +3%.We concluded that, the fusion technique revealed the complementary nature of both modalities in detecting features that highly correlated to mental stress.

Figure 1 .
Figure 1.The setup of multi-modal fNIRS-EEG experiment for subject performing arithmetic task.

Figure 2 .
Figure 2. Experimental protocol and task designed sequences.

2 O
Hb represents the segmented oxygenated hemoglobin signal and N is the length of the signal.

Figure 3 .
Figure 3. Topographical map of oxygenated hemoglobin concentration under control condition for average of 12 subjects.Red colour indicates high concentration levels of oxygenated haemoglobin and blue colour indicated less level of oxygenated haemoglobin concentration.

Figure 4 .
Figure 4. Topographical map of oxygenated hemoglobin concentration under stress condition for average of 12 subjects.Red color indicates high concentration level of oxygenated hemoglobin and blue color indicated less oxygenated hemoglobin concentration.

Figure 5 .
Figure 5. Normalized alpha rhythm power in all the recorded electrodes position (PFC area) for average of 12 subjects.Blue colour indicated the average alpha power under control task and red colour indicated the average alpha power under stress task.

Figure 6 .
Figure 6.Boxplot of the classification accuracy obtained using individual and combined modalities.