Decision Fusion of EEG and fNIRS Signals

 Abstract —In this study, we investigated the use of multimodal functional neuroimaging in detecting mental stress on the prefrontal cortex (PFC). We recorded Electroencephalography (EEG) and functional Near Infrared Spectroscopy (fNIRS) simultaneously from 20-subjects performing mental arithmetic task under control and stress conditions. Stress was induced in this study based on two established stressors – time pressure and negative feedback about peer performance. We explored decision fusion by using support vector machine classifier for each modality, and optimizing the classifiers based on Receiver Operating Characteristic (ROC) curve values. Experiment results revealed significant reduction in alpha rhythm and mean change in concentration of oxygenated hemoglobin at PFC when stressed, p< 0.001 and 0.0001 respectively. The decision fusion improved significantly the detection rate of mental stress by +7.76% and +10.57%, when compared to sole modality of EEG and fNIRS, respectively.

 Abstract-In this study, we investigated the use of multimodal functional neuroimaging in detecting mental stress on the prefrontal cortex (PFC). We recorded Electroencephalography (EEG) and functional Near Infrared Spectroscopy (fNIRS) simultaneously from 20-subjects performing mental arithmetic task under control and stress conditions. Stress was induced in this study based on two established stressorstime pressure and negative feedback about peer performance. We explored decision fusion by using support vector machine classifier for each modality, and optimizing the classifiers based on Receiver Operating Characteristic (ROC) curve values. Experiment results revealed significant reduction in alpha rhythm and mean change in concentration of oxygenated hemoglobin at PFC when stressed, p<0.001 and 0.0001 respectively. The decision fusion improved significantly the detection rate of mental stress by +7.76% and +10.57%, when compared to sole modality of EEG and fNIRS, respectively.

I. INTRODUCTION
TRESS is a growing problem in our society and is part of our daily life. We spend most of our time as adult at workplace. Extended periods of high workload and time pressure contribute to increased level of stress [1] [2]. Conventionally, mental stress is evaluated by using subjective questionnaires, cortisol level and other physiological signals (i.e. heart rate, blood pressure and skin conductance) [1]. Such methods known to be affected by circadian rhythm humidity and cardiovascular diseases [3,4]. As stress is originated from the amygdala, it is desirable to study the stress effects on brain activities and measure them with modern neuroimaging modalities. However, each of the modality has its own limitations.
Medical imaging modalities such as functional magnetic resonance imaging (fMRI) and positron emission tomography (PET) have good spatial resolution but have limitations in term of temporal resolution and susceptibility to movement artefacts [5]. Moreover, subjects need to remain still during fMRI/PET measurements [6]. Electroencephalography (EEG) is a possible alternative neuroimaging modality as it has a temporal resolution in the order of a few milliseconds, which makes it suitable for measuring dynamic cortical changes during workplace activities [7].Unlike functional magnetic resonance imaging and positron emission tomography, modern EEG hardware is light-weight and portable enough to be used during unconstrained full-body motion. Studies have shown that EEG signals can be used to classify mental stress from resting state [8,9]. Nevertheless, EEG has traditionally been thought of as possessing poor spatial resolution and being highly prone to motion artifacts [10]. Functional Near-Infrared Spectroscopy (fNIRS) detects the neural activities of the brain by measuring changes in the concentration of oxygenated hemoglobin (O2Hb) and deoxygenated hemoglobin (HHb) in the cortex [11,12]. The fNIRS is a promising alternative, achieving some middle ground in spatial and temporal resolution as well as mobility between EEG and fMRI techniques [13][14][15][16]. Additionally, the measurements have been shown to be consistent with fMRI and EEG [17,18]. In this paper, we hypothesize that fusion of EEG and fNIRS can improve the detection rate of mental stress from using either modality alone. Compare to traditional neuroimaging, EEG+fNIRS have the advantages of allowing human cortical activities to be measured without major constraints on how subjects are positioned (mobility), and complement each other in the temporal and spatial resolution. Recently, there has increased interest in the fusion of neuroimaging modalities. Fusion is a method of gathering and analyzing complementary modalities utilizing their common and unique features to provide better understanding in the detection and treatment of brain diseases. In the fusion terminology, data from different modalities usually analyzed either in the feature level or at the decision level [19,20]. In the feature level, fusion techniques incorporate all features from different modalities into a combined analysis to explore associations across them through variations across subjects. Joint independent component analysis (jICA) and canonical correlation analysis (CCA) are of the most common feature-fusion techniques [19]. In the decision level, each data are analyzed separately to provide a local decisions that are based on individual features. The local decisions then combined using decision fusion technique to improve the overall classification performance [21]. Decision fusion has the advantages of using the most suitable methods for analyzing individual modality, which provide much

Decision Fusion of EEG and fNIRS Signals
Fares Al-Shargie S flexibility than feature-level fusion. Additionally, decisionlevel fusion offers scalability in term of the modalities used in the fusion process which is difficult to achieve in the featurelevel fusion [22]. This paper introduced a decision-level fusion to fuse two decisions according to the operating points on their receiver operating characteristic (ROC) curves. Our experimental results demonstrate that, the proposed decision fusion achieves significant improvement over individual EEG/fNIRS classification performance. The paper is organized as follows. In section II, we briefly describe on the participants selection, development of the stress task, and describe the sets of EEG and fNIRS signals used. We explained the methods used to process the data and the features used for the statistical analysis, classification and fusion. In section III, we present the results of individual modality, the statistical analysis and the results of fusion. In section IV, a discussion of the presented results is provided in the light of existing studies. Finally, in section V, we conclude the paper.

A. Participants
Participants were twenty-eight healthy right-handed students from the American University of Sharjah (22±2 years, 13 males; 21±1.5 years, 15 female). All of them reported normal or corrected-to-normal vision. None of the participants had a history of neurological or psychiatric illness and had no current or prior psychoactive medication use. They were asked to abstain from caffeine, exercise, energy drink and tobacco use for 24 hours before testing. All participants were explained about the study and gave informed consent before the start of the experiment. The experiment protocol conducted in accordance with the Declaration of Helsinki and approved by local ethics review committee at the American University of Sharjah.

B. Emotion-Eliciting Pictures
participants. Participants first practice the task for five minutes and the average time taken for each individual in answering the questions was recorded. indicators (one for the participant's performance and one for the averaged peer performance fixed at 90% accuracy) were displayed on the computer monitor to further induce stress in experiment participants. Fig. 1. Schematic of the experimental procedure for mental stress study. Two recordings were performed in this experiment; one for control condition and the other one for stress condition. In each recording, there were five blocks. In each block, mental arithmetic was allocated for 30 s followed by 20 s rest.

C. Data Acquisition
Electrophysiological signals were recorded using 64 Ag/AgCl scalp electrodes arranged according to the standard 10-20 system (ANT waveguard system and ASA Lab 4.9.2 acquisition software, ANT Neuro, the Netherlands), sampled at 500 Hz. The impedances of all EEG electrodes were maintained below 20 kΩ throughout the recording.

D. Data processing
EEG signals were preprocessed using custom script as well as the EEGLAB toolbox(9.0.4) [25]. The EEG signals were rereferenced using the common average reference. Baseline removal and DC offset was done by subtracting the mean from the data. All EEG signals were band-pass filtered using FIR filter with 0.5 Hz and 30 Hz cut-off frequencies. The electrical power line noise was removed on ICs using the CleanLine plugin of EEGLAB. Eye-blinks and eye-movements were removed manually by visual inspection as well as using Independent component analysis (ICA) technique available in EEBLAB. The components representing artifacts, such as eye blink, eye movements, and muscular activities were removed and the remaining components were used to reconstruct the clean EEG signals.

E. Statistical analysis
To reveal the differences in brain responses between control and stress groups in the subjective as well the simultaneous measurement of EEG+fNIRS, we analyzed the differences between them in channel/electrode basis using two-sample ttest. In each electrode/channel, the differences were considered statistically significant if the p value is less than 0.001, p< 0.001.

F. Classification performance
To classify EEG and fNIRS signals, we used support vector machine (SVM) classifier. The SVM is a supervised machine learning technique widely used for classification, regression and density estimation [31]. SVM is selected for its ability to model linear as well as more complex decision boundaries. LIBSVM software was used to build the SVM classifier and employed radial basis function (RBF) kernel to nonlinearly map data onto a higher dimension space [32].
A 10-fold cross-validation scheme with randomization was applied to each feature vector. In the 10-fold cross validation, each of the EEG and fNIRS features were split into ten subsets. Nine subsets were used to train the SVM classifier, and the remaining one subset was used for estimation of classification accuracy, sensitivity and specificity. This procedure was repeated ten times with each subset having an equal chance of being the testing data. Then we analyzed the classifier performance by using receiver operating characteristic (ROC) curves, which plot the sensitivity versus (1 minus the specificity) with varies threshold. We also evaluated the area under the ROC curve (AUC) as a measure of a classifier's discriminatory power.

G. Decision fusion of EEG and fNIRS signals
The decision fusion was achieved by fusing the outputs/ decisions from two classifiers, one for EEG signals and the other for fNIRS signals. Each classifier was calibrated based on the operating points of EEG and fNIRS receiver operating characteristic (ROC) curves. The type of classifiers used in this study is support vector machine (SVM) with radial basis functions.
Output by individual classifier is denoted as uk, (EEG: k = 1; fNIRS: k = 2), where uk = 0, if the decision of kth classifier is H0 (no-stress) and uk = 1 if the decision of classifier is H1 (stress). The decision of classifier uk depends on the features vectors xk, k = 1,2.
where α1 and α2 are sets of threshold value of EEG and fNIRS classifiers respectively. The performance characteristics of classifier k specified by P(uk=1|Hj) where P(uk=1|H0) = Pfk is the probability of false positives, and P(uk=1|H1) = Pdk is the probability of true positives. Using these probabilities, the likelihood ratio value of a binary decision variable may be obtained by: The decision at the fused level depends only on local decisions and their probability of true positive Pdk and false positive Pfk.
Since individual classifiers are based on different modalities, the two decisions assumed to be statistically independent. The fusion likelihood ratio could then be generated using the following form: The optimal fused decision rule uses the fusion likelihood ratio as a classification decision variable and then compares it to threshold β for decision u according to the following: where H is the Heaviside function. By varying the threshold β value (corresponding to different decision rules), N number of operating points were derived for the fused ROC curve. The probability of true positive rate and false positive rates for decision-fused classifier operating points could then be calculated based on the following equations: The entire decision-fused ROC curve obtained by deriving a series of Pk,fusion(β) using multiple combinations of operating points on EEG-/fNIRS-based classifier ROC curves.

III. RESULTS
Experiment result showed that subjects' performance decreased with stress, accuracy score of answering the task correctly was <40%, as expected. The results of subjective ratings of workload measured by NASA-TLX across the control and the stress conditions for all subscales is summarized in Table 1. Additionally, the overall workload of the NASA-TLX is also presented with their standard error. The weighted workload (WWL) in the control and stress conditions were 22.2 and 75.4, respectively. Based on the participant's subjective responses, in control condition own performance (OP) and in stress condition frustration (FR), temporal demand (TD) and physical demand (PD) had dominant importance and in control and stress conditions, effort (EF) had the lowest importance. Overall, there was a significant differences between the subjective score from the control to the stress condition across all the subjects and the NASA-TLX subscales. The t and p values of the subjective evaluations are summarized in Table 1. The result confirms the inducement of moderate stress level in every subject. The result of EEG alpha rhythm revealed significant decrease from control to stress condition, p<0.001. The decrease was found across all the electrodes in all the subjects. The overall statistical analysis (t and p values) are summarized in Table 2, with F4, F8 and Fp1 showing the highest t-values. Figure 3 shows the normalized alpha rhythm under control and stress conditions as averaged of all electrodes on the PFC represented by the box plots. Similarly, the results of fNIRS demonstrated significant increase in the O2Hb changes associated with a decrease in the HHb during the control arithmetic task relative to the pre-task baseline over the entire PFC (except Ch-18 and Ch-23; in the case of O2Hb only) . Likewise, under stress condition the O2Hb changes decreased in association with an increase in the HHb in most of the PFC regions relative to pre-task, on average (except at Ch-7 and Ch-12). The overall grand average timecourse of O2Hb (red line) and HHb (blue line) changes at control and stress conditions are shown in Fig.4 and Fig.5, respectively. As the O2Hb highly reflect the cortical activities we limited our analysis in this work to their response.
As the focus of this study is to investigate on the behavioral responses of the neural activities under neural-control and stress conditions, we conducted the statistical analysis between the two groups. The entire study, revealed significant decreases in the O2Hb changes from control condition Fig.4 to stress condition Fig.5 over some regions of the PFC areas (excluded at Ch-7, Ch-12, Ch-17, Ch-18 and Ch-23). As can be seen from Fig.4 and Fig.5, stress is a region specific than diverse and showed the left PFC area is the less affected region by stress. Table 2 summaries the statistical analysis of all the subjects and channels represented by t and p values with their standard deviations (SD). The lowest standard error across all channels is located at the right ventrolateral PFC, Ch-13, Ch-14 and Ch-19, indicating that this sub-region is dominant across the subjects to mental stress.   The classification result of individual modality and after fusion is shown by their ROC curves. Features from the seven electrodes (FP1, FP2, Fz, F3, F4, F7 and F8) were used for evaluating EEG modality and for fusion with the fNIRS1 and fNIRS2. The fNIRS(1,2) channel selection was based on twolateralize neighboring channels in each subregion of the PFC areas (within the right and left: DLPFC; VLPFC and FPA). The first fNIRS1 lateralize channels were Ch-2, Ch-4, Ch-3, Ch-23, Ch-13, Ch-22 and Ch-20. The second fNIRS2 lateralize channels were Ch-2, Ch-1, Ch-7, Ch-18, Ch-19, Ch-15 and Ch-16. Fig.6 and Fig.7 show the ROC curves of individual EEG, fNIRS1, fNIRS2, fusion of EEG+fNIRS1 and fusion of EEG+fNIRS2. The black line shows the result from fNIRS(1,2), blue line shows the result from EEG and the red line shows the result of fusing EEG+fNIRS(1,2), respectively.
The average classification performance of sole EEG, sole fNIRS1, fusion of EEG+fNIRS1, sole fNIRS2 and fusion of EEG+fNIRS2 in the form of accuracy, sensitivity, specificity and area under the ROC curves are summarized in Table 3. Decision fusion shows improvements in the classification accuracy in the range of +7.76% to +10.57% compare to sole EEG and sole fNIRS(1,2) respectively. Similar improvements were found in the classification sensitivity, specificity and in the area under ROC curves in the range of +3.45% to +11.65%. Using two sample t-test, fusion of EEG+fNIRS(1,2) significantly improves the overall classification performance in all the metrics with p<0.001.

IV. DISCUSSION
In the present study, we investigated the performance of fusing simultaneous measurement of EEG and fNIRS signals for discriminating between set of task at control and stress conditions. EEG provides data on electrical cortical activity whereas fNIRS monitor the cerebral blood flow and the oxygen metabolism on the cortex. The two measurement systems complement each other in quality (data sources) as well as in the quantity. Thus, integrating this physiological responses enhance the detection rate of mental stress. The stress stimuli used in this study is based on arithmetic task complexity with time pressure and negative feedback of the subject performance. The task have proven to induce stress on the participants, as proven in our previous study in which the salivary cortisol significantly increased from control to stress condition in all the participants [23]. Additionally, we involved the behavioural responses as well as the subjective evaluation collected using NASA-TLX and demonstrated that all participants experience stress while solving the arithmetic task under time pressure.
The EEG results showed that stress disrupt the cortical activities within the entire PFC region. There was a significant reduce in alpha rhythm from control to stress condition with mean p-value of p<0.001. The reduce in alpha rhythm in this study could be explained as that, the time pressure with negative feedback induced negative emotion which with continuous recording time further increase the level of stress. Similar decrease in alpha rhythm has been found in previous emotional and anxiety studies [33][34][35][36]. Similarly, fINRS results demonstrated that stress reduced the cortical activity over the right ventrolateral PFC, DLPFC and right frontopolar areas. There was a significant decrease in the oxygenated hemoglobin concentration in these regions with less standard error on the right ventrolateral PFC. The less error at this particular region indicates that all participants perceived the same level of stress equally and suggest it as the most sensitive region to stress exposure.
Using support vector machine (SVM) as an individual classifier, we were able to classify brain activities under stress from that of control state significantly. The performance of SVM classifiers were evaluated based on accuracy, sensitivity, specificity and the area under ROC curve. Using features from EEG only gives a mean classification of 88.69%, 87.6%, 89.7% and 95.1% accuracy, sensitivity, specificity and area under ROC respectively. Similarly, the result obtained from the first and second lateralization channels of fNIRS(1,2) showed on average 85.88%, 83.45%, 88.35% and 92.95% accuracy, sensitivity, specificity and area under ROC curve respectively. Decision fusion of multiple operating points of EEG+fNIRS(1,2) demonstrated significant improvements in the detection rate of mental stress with 96.45%, 95.10%, 97.70% and 98.55% accuracy, sensitivity, specificity and area under ROC respectively. The percentage of improvements in the detection rate of mental stress was in the range of 3.45% to 11.65% in all the four metrics namely; accuracy, sensitivity, specificity and area under the ROC curve. This result support the assumption of that EEG and fNIRS provide complementary information for better stress detection.
Up to this date, only few studies have looked into the possibilities of combining hemodynamic responses with their electrophysiological counterparts in a hybrid method aiming at improving the overall systems performance [20,[37][38][39][40][41][42]. However, these studies are based on motor imagery tasks which is relatively simple and stable. Additionally, these studies used large number of EEG and fNIRS channels which take longer time for preparation, discomfort and reduce the portability. For example, Fazli et al. proposed a hybrid sensory motor rhythm combined 24 fNIRS channels and 37 EEG Electrodes. The researcher used a meta-classifier to combine the output probability of the individual classifier-modality. The results showed that the simultaneous measurement of the EEG+fNIRS can significantly improve the classification accuracy of the motor imagery by an average of +5% [20]. Lee et al. [38] used similar number of channels/electrodes and demonstrated that bimodal EEG+fNIRS signals increase the classification performance by 10% for the motor imagery task. Morioka et.al [42] used 49 channels and 64 EEG electrodes and reported that the hybrid system of EEG+fNIRS improved the spatial attention decoding accuracy by 8%. Blokland et al. [43] recently examined the principle of combining these modalities in patients with tetraplegia. The researcher used 2 channels with 8 electrodes and reported an improved accuracy of 1% in the brain switch control for some subjects. The highest improvements of 11.65% supports the flexibility of our experiment and the robustness of the proposed fusion technique.

V. CONCLUSION
In this study, we investigated if decision fusion level of EEG and fNIRS signals could improve the detection rate of mental stress on the PFC. The experiment results showed that stress impaired PFC activities, specifically the right ventrolateral PFC area. The proposed fusion technique significantly improved the classification accuracy of mental stress by +10.57%, and +7.76% compared to sole fNIRS and sole EEG, respectively. Similar improvements were found in the sensitivity, specificity and area under ROC in stress detection. Our study showed that, albeit with less number of electrodes/channels, the improvements of fusion were significant, p<0.001.