Neural network in the wheelchair control system for severe disabilied people using eeg signal and camera

ata before putting in multilayer neural network to classify these partterns. Combining EEG signals and camera processing aim to identified and classified process more easily and effectively. 1.5 Object and scope of the research The main object in thesis is to use multilayer neural network to classify 5 EEG signal partterns into control commands corresponding to 5 commands to control the wheelchair as: Forward, Turn right, Turn left, Reverse and Stop. In thesis also mention to image processing to detect the eye direction aim to help the system work more easily and effectivly. However, the thesis don’t focus more on image processing, but on EEG signal processing. 1.6 The contribution of thesis 1.6.1 The contribution about theory Find out the observation board which easy to use to collect data, combine scientifically between feature extraction algorithm and cluster data before putting into neural network to classify data partterns. 1.6.2 Practical contribution The experimental result of thesis show that the EEG signal partterns classification through eye observation (with differnce image partterns), for people who has mind and eyes as normal people could absolutely performance. CHAPTER 2 - THEORETICAL BASIS 2.1 EEG signal and its characterizations Delta wave (0 – 3 Hz), the highest amplitude as figure 2.1, it often appear at the child up to 1 year old and adult when sleeping, well sleep. It represents the grey matter of the brain. This wave usually appears everywhere on the scalp. Trang 5 Figure 2.1 The Delta wave The Theta wave (3 – 7 Hz), it appear when the eye are closed and the mind is in a relax state as Figure 2.2, it apperas in adult or when awake in the elderly and often appears in the temples. Figure 2.2 The Theta wave The Alpha wave (7 – 13 Hz), it appear often in elder as Figure 2.2, Alpha wave usually appear on both sides of scalp but having an uneven amplitude, this waves appear when the eye are closed (a state of relaxation) and often disappear when the eyes are opened or under stress. Figure 2.3 The Alpha wave The Beta waves (13 – 30 Hz), with low amplitute as Figure 2.4, This waves usually appears in patients who are often in a state of alertness, prevention, and anxiety this waves are distributed symmetrically on both sides and most clearly at the front, it usually appear in front and at the top of the cerebral cortex, the amplitude is less than 30uV. Figure 2.4 The Beta wave Trang 6 The Gamma wave (30 – 45 Hz), it often referred to as fast Beta wave. This wave usually has low amplitude and rarely appears, but the discovery of this wave plays a important role in identifying neurological diseases that occur in the center of the cerebral cortex. Figure 2.5 The Gamma wave 2.2 The electrode positions on the scalp The brain is one of the largest and most complex organs in the human body, it is made up of over 100 billion nerves, communicating with 1,000 billion synapses. The electrode positions are mounted on the scalp according to international standard 10/20 as Figure 2.6. Figure 2.6 Electrode positions according to international standard 10/20 CHAPTER 3 CONTROL MODEL CONTRUCTION To begin the research process, author used the database provided on the prestigious University of San Diego (UCSD) website of the USA, ranked 38 in the world in 2018, this data was obtained from participants when looked at 5 different image objects (human, city, landscape, flower and animal), with 8 color bits and size (256 pixels Trang 7 wide and 384 pixels high), the total number of samples is 21,000. In this chapter, author present to built the model from simple to complex step by step, and then evaluate the experimental results on database provided by the University of San Diego (UCSD), and from 80 students from HITU to make clarify the contribution and scientific meaning of the thesis. 3.1 Single-Layer neural network model At first, author built a single neural network to separate two parttern of signals (animals and not animals), the purpose of this study is to evaluate whether the neural network meets the classification requirements, thesis used Matlab software for this experimental process. The features of EEG database is extracted by Wavelet transform (Mexico hat) and used a single neural network to identify. The system model is shown in Figure 3.1, this model included of 2 stages: stage 1: preprocessing raw data signals and then synthesize into 5 basic EEG signals Delta, Theta, Alpha, Beta and Gamma. Stage 2: builting a single neural network with 5 inputs corresponding to 5 basic EEG signals: Delta, Theta, Alpha, Beta, Gamma and one output to determine the clasified results. Figure 3.1 Single-Layer neural network model The training process is performed on the database with the following parameters:  Learning rate: 0.7. Raw data Classified result Pre-processing Single-Layer Neural Network Trang 8  Initial random weight in the range from -0.5 to 0.5  The error threshold is 1x10-5 based on MSE (Mean Square Error).  The maximun number loop: 5.000. Experimental results of identification on the database are shown in Table 3.1. Table 3.1 Experimental results on database Image types Animal/Landscape Identification Rate France Landscape 99,13% Wild sheep Animal 98,67% Wild cats Animal 99,28% Bali, Indonesia Landscape 62,44% Wild animals Animal 99,64% California Coasts Landscape 56,89% Wolves Animal 98,64% Mushrooms Landscape 95,16% Kenya Animal 99,76% The big Apple Landscape 98,79% Snakes, lizards... Animal 98,32% Caves Animal 67,18% Polar bears Animal 99,03% Exotic Hong Kong Landscape 98,72% Images of France Landscape 99,37% Fabulous fruit Landscape 98,25% Wild animals Animal 93,97% Sand & solitude Animal 98,42% Lions Animal 62,78% Trang 9 Image types Animal/Landscape Identification Rate Great Silk Road Landscape 98,47% From the experimental results in Table 3.1, author found that the average accuracy rate of the identification results on the database was 91.15%. 3.2 Multi-Layer neural network model 3.2.1 System Model Based on the results achieved from the single neural network model, author continued to develop a multi neural network model with the results of classifying 5 EEG signal partterns corresponding to 5 control signals with accurate rate 93.57%. Table 3.2 describes the result of 05 control commands corresponding to the equivatent image types. This model uses Wavelet transform to noise signals and extract features, then using K-mean algorithm to cluster the characteristics of the signal partterns and then put into the multi-layer neural network to classify, in this model, author chooses 10 channels to reduce processing time and enhance performane. System model is shown in Figure 3.2. Figure 3.2 Multi-Layer neural network system model EEG Signal Selecting Channel Wavelet Transform Clustering Muli-Layer Neural Network Classified Result Trang 10 3-layer neural network model is shown in Figure 3.3. The first layer contains five nodes which are Delta, Theta, Alpha, Beta and Gamma, these classes is called the input layer. The second layer is the hidden layer, the number of nodes in the hidden layer is set to 5, 10, 15, 20, 25, 30, 35, 40, 45 and 50. The output layer contains a node, the result of this node is used to classify the EEG signal. The activation function used in this model is hyperbolic tangent, the value of the output is in the range [-1, 1]. Figure 3.3 Multi-Layer neural network model Before using the model, the neural network needs to training stage. The training algorithm is shown in figure 3.4, the model used backpropagation algorithm. Trang 11 Figure 3.4 Neural network training algorithm 3.2.2 Experimental results with multi-layer neural network model The dataset consists of 21,000 samples divided into subsets for training (70%), validation (15%) and testing (15%). The system uses Matlab and EEGLab tools for the testing process, the neural network is divided into two test stages. The training stage is performed on the training database, using structures with different hidden nodes in the hidden layer of neural network, with the following parameters as follows:  Learning rate: 0.7.  Initial random weight in the range from 0 to 1  The error threshold is 1x10-5 based on RMSE (Root Mean Initial random weight Get value Delta, Theta, Alpha, Beta, Gamma Calculate the value of nodes in hidden layer Calculate the ouput value of nodes in the hidden layer Calculate the input value of the output nodes Calculate the output value of the output nodes Calculate the error of the output layer Calculate the error of the hidden layer Calculate the error of the system Error system <= threshold value? End Update weight Begin True False Trang 12 Square Error).  The maximun number loop: 5.000. The accuracy of classification is measured by the ratio of result of wrong classification to the total number of samples follow formula (3.1). = − (3.1) Where n is the total number of samples, ntrue is the number of samples with correct classification results. Table 3.2 Experimental results Number Nodes in hidden layer Average Error Minimum Error Accurate Rate 5 25.21% 21.98% 78.02% 10 23.77% 20.04% 79.96% 15 20.44% 17.13% 82.87% 20 17.76% 14.21% 85.79% 25 15.43% 11.88% 88.12% 30 12.98% 10.06% 89.94% 35 9.87% 7.74% 92.26% 40 7.74% 6.43% 93.57% 45 9.56% 7.92% 92.08% 50 10.24% 8.63% 91.37% Observing in Table 3.2 and see that the classification results of the database are constantly increasing until the best possible value (40 Nodes in the hidden layer and the accurate rate is 93.57%), After that, the result values begin to decrease as the number of nodes in the hidden layer increases, This is called overfitting. A confusion matrix contains about actual classifications and predictions made by system. Table 3.3 describes the performance of the system evaluated by the database in the matrix confused with 40 nodes in the hidden layer. Table 3.3 Matrix confusion of classification results Trang 13 Actual Pred ictio n Anima l Lands cape City Huma n Flower Animal 93.8% 1.4% 1.5% 2.0% 1.6% Landscape 1.1% 93.6% 3.1% 1.5% 1.2% City 1.6% 1.3% 93.5% 1.6% 1.8% Human 1.9% 1.8% 1.3% 93.2% 1.7% Flower 1.7% 1.9% 0.8% 1.8% 93.8% To provide a more intuitive and easy-to-understand about result of predictive quality, the following formulas are used for effective quality test. Accuracy (AC) is an accurate prediction rate. It is determined using the formula (3.2). = + + + + (3.2) Precision (P) is the ratio of correct times that are predicted to be accurated, calculated using the formula (3.3). = + (3.3) Where, True Possitive (TP) refers to the correct database that are correctly classified to be true. True Negative (TN) refers to incorrect database that are incorrectly classified to be false. False Possitive (FP) refers to incorrect database that are incorrectly classified to be true. False Negative (FN) refers to incorrectly categorized database to be false. The rate of identification with 40 hidden nodes in the hidden layer is given in table 3.4 Table 3.4 Experimental results TP TN FP FN AC P Animal 93,8% 93,6% 6,4% 6,2% 93,7% 93,6% Landscape 93,6% 93,2% 6,8% 6,4% 93,4% 93,2% City 93,5% 93,8% 6,3% 6,6% 93,6% 93,7% Trang 14 Human 93,2% 93,4% 6,6% 6,8% 93,3% 93,4% Flower 93,8% 93,9% 6,1% 6,2% 93,9% 93,9% These results are also compared with previous studies such as determining EEG based on winking with 15,360 samples and reaching 90.85%, with decision tree reach the rate 85%, based on eye movement by 2 experiments with 3,600 samples and 8,320 samples reaching acuracy rate 85%. 3.3 Design of synthetic model for signal processing This model was developed from a multi-layer neural network model in section 3.2. Beside of identifying the EEG signal, it also combined the user's eye direction signal through the camera, this model focuses on the items as below:  Converting EEG signals using Hilbert Huang Transform (HHT) method to reduce noise signal because HHT conforms to EEG signal and for better results than other transform methods.  Eye direction recognition based on the user's face image combined with EEG signal recognition to improve the effectiveness of the identification.  Design a system that includes hardware and software for testing in realtime.  Experimental data was collected from 80 students of HITU who volunteered to participate. The system consists of 3 blocks as shown in Figure 3.5. Trang 15 Figure 3.5 Synthetic system model  The first block is the EEG signal recognition block to extract 5 features.  The second block is the eye direction recognition block by recognizing the eyes and eyebrows from the user's face image to extract 4 features.  The third block is a multi-layer neural network with 9 input nodes (4 for camera and 5 for EEG), 5 output nodes are classified equivalent 5 control commands as “FORWARD”, “TURN RIGHT”, “TURN LEFT”, “REVERSE” and “STOP”. 3.3.1 EEG signal recognition block This block behaves like the model in section 3.2, at first, selecting useful information channels and remove channels with redundant information, next, using HHT to extract feature and eliminate noise signals, and then, use the K-Means algorithm to cluster data. 3.3.2 Identifing eye direction signal Face images are received from the camera and are cropped in the area of eye information to reduce processing time, after that, the face image is converted into a binary image that satisfies the requirements Face image Converting to binary image Detecting eye and eyebrow Rating between eye and eyebrow Extracting 4 features EEG signals Selecting channels Hibert Huang Transform Clustering Extracting 5 features MULTILAYER NEURAL NETWORK Trang 16 for showing both eyebrows and eyes as shown in Figure 3.6. Threshold values are calculated by isodata algorithm. Figure 3.6 Detect eyes and eyebrows Using image segmentation algorithm to detect the center of the eye, the segment of eye and eyebrows as Figure 3.7. Figure 3.7 Pupil, segmented eyes and eyebrows Calculate the ratio of left eye and left eyebrow according to formula (3.4) as Figure 3.8 = (3.4) Figure 3.8 The ratio of left eye and left eyebrow Calculate the ratio of right eye and eyebrow according to formula (3.5) as Figure 3.9. = (3.5) Trang 17 Figure 3.9 Right eye ratio and right eyebrow Calculate the percentage of pupil center of the left eye and the length of the left eye by the formula (3.6) as Figure 3.10. = (3.6) Figure 3.10 Rate of pupil and eye length Calculate the proportion of the pupil center of the right eye and the length of the right eye according to formula (3.7) as Figure 3.10. = (3.7) 3.3.3 Multilayer neural network model backforwards The backpropagation multi-layer neural network model consists of 3 layers as Figure 3.11. Trang 18 Figure 3.11 Multi-layer neural network model  The first layer contains 9 nodes like Delta, Theta, Alpha, Beta, Gamma, d1, d2, d3 and d4. This class is called the input class.  The second layer is the hidden layer, the number of nodes in the hidden layer is 11 nodes.  The output layer contains 5 nodes, the result of this node is used to classify EEG signals. Because the activation function is hyperbolic tangent, the value of the output node is between [-1, 1]. Since the output has 5 nodes, which one has the largest value, it will be selected and that is the control signal. 3.3.4 Select the data set and experimental results Experimental data was collected from 80 volunteering students of Ho Chi Minh City College of Industry and Trade (HTIU). Students wear the Emotiv EEG device and sit 120 cm away from the observation board. Experimental data is divided into 3 data sets as follows:  Training data set was collected from 70% data of 60 students. 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 10 11 1 2 3 4 5 Input Layer Hidden Layer Output Layer Trang 19  The first test data set was collected from 30% remaining data of 60 student.  The second set of test data was collected from the remaining 20 students. After training the neural network from the training database, experimental results of the first test data set are shown in the confusion matrix, shown in Table 3.5 and identification rate in Table 3.6. Table 3.5 The confusion matrix of the result from the first database classification Actual classification Predic tive Classi ficatio n Huma n Anim al Flowe r City Lands cape Human 91,2% 1,6% 3,1% 0,9% 2,7% Animal 1,9% 91,1% 1,2% 3,2% 2,0% Flower 2,8% 2,5% 92,8% 1,7% 1,5% City 2,4% 2,1% 0,7% 92,1% 1,9% Landscape 1,7% 2,7% 2,2% 2,1% 91,9% Table 3.6 Experimental results on the first database TP TN FP FN AC P Human 91.2% 91.3% 8.7% 8.8% 91.3% 91.3% Animal 91.1% 91.1% 8.9% 8.9% 91.1% 91.1% Flower 92.8% 92.6% 7.4% 7.2% 92.7% 92.6% City 92.1% 92.2% 7.8% 7.9% 92.2% 92.2% Landscape 91.9% 92.1% 7.9% 8.1% 92.0% 92.1% The experimental results of the second test database are shown in the confusion matrix in Table 3.7 and the identification rate in Table 3.8. Trang 20 Table 3.7 Confusion matrix of classification result for the second database Actual classification Predic tive Classi ficatio n Human Anim al Flowe r City Lands cape Human 90.7% 1.9% 2.7% 1.1% 3.1% Animal 1.4% 90.8% 2.1% 2.6% 2.4% Flower 2.3% 2.3% 92.3% 3.2% 1.7% City 3.1% 2.6% 1.2% 91.6% 1.3% Landscape 2.5% 2.4% 1.7% 1.5% 91.5% Table 3.8 Experimental results on the second test dataset TP TN FP FN AC P Human 91.2% 91.3% 8.7% 8.8% 91.3% 91.3% Animal 91.1% 91.1% 8.9% 8.9% 91.1% 91.1% Flower 92.8% 92.6% 7.4% 7.2% 92.7% 92.6% City 92.1% 92.2% 7.8% 7.9% 92.2% 92.2% Landscape 91.9% 92.1% 7.9% 8.1% 92.0% 92.1% Experimental results for eye direction signals are shown in Table 3.9 and for EEG signals shown in Table 3.10. Table 3.11 compares 3 experimental results. The graph in Figure 3.12 shows the chart of two signals when identifying them separately. Table 3.9 Experimental results on eye direction signals TP TN FP FN AC P Human 85.1% 85.7% 14.3% 14.9% 85.4% 85.6% Animal 84.5% 84.1% 15.9% 15.5% 84.3% 84.2% Flower 87.3% 86.3% 13.7% 12.7% 86.8% 86.4% City 83.6% 84.0% 16.0% 16.4% 83.8% 83.9% Landscape 84.2% 83.2% 16.8% 15.8% 83.7% 83.4% Trang 21 Table 3.10 Experimental results on EEG signals TP TN FP FN AC P Human 90.2% 89.9% 10.1% 9.8% 90.1% 89.9% Animal 90.3% 90.0% 10.0% 9.7% 90.2% 90.0% Flower 92.3% 91.8% 8.2% 7.7% 92.1% 91.8% City 90.7% 90.4% 9.6% 9.3% 90.6% 90.4% Landscape 90.4% 90.5% 9.5% 9.6% 90.5% 90.5% Table 3.11 Experimental results of 3 identification methods EEG and Camera EEG Camera Human 90,8% 90,1% 85,4% Animal 90,9% 90,0% 84,3% Flower 92,2% 91,8% 86,8% City 91,7% 90,4% 83,8% Landscape 91,7% 90,5% 83,7% Figure 3.12 Experimental results on 3 separate methods Trang 22 CHAPTER 4 CONSTRUCTING SOFTWARE AND HARDWARE TO CONTROL THE WHEELCHAIR MODEL In this chapter, author built the wheelchair hardware and software system based on the models presented in Chapter 3. Then evaluate the experimental results to clarify the practical contribution of the thesis. 4.1 Wheelchair control software system Wheelchair control software is designed on Visual Studio C# 2015, the software interface includes 4 functions as follows: 4.1.1 Login to the system In this section, it is mandatory for anyone who uses software to have an account to log in to the system, personal account used to manage the database of brain signals, training time, accurate rate during the control, the interface of the login part in Figure 4.1. Figure 4.1 System login interface 4.1.2 Wheelchair control training For a person who has never participated in control, this step must be done, as well as someone who has never drive, they had to learn to drive. In order to control the wheelchair, the participants have to control 5 commands with the acuary rate of more 90% for each command. The purpose of this training is to help participants become familiar with wheelchair control and concentrate in control. The training software interface in Figure 4.2. Trang 23 Figure 4.2 Training software interface 4.1.3 View the EEG signal via graph The software also has the function of reviewing the graph of EEG signals for every channel, depending on the purpose of the research, just click on the electrode channel position as shown in Figure 4.3. Figure 4.3 Viewing EEG signals via graph 4.1.4 Extracting the feature of EEG signals To extract the feature of EEG signals, author used the HHT algorithm as presented in the theoretical basis, total number of analyzied channels is 10, each channel is analyzed into 12 IMF (intrinsic functions), so we have all 120 IMFs for each processing. The program extracted 1 signal channel into IMFs, and then from these IMFs extracted in 5 basic waves as Figure 4.4. Trang 24 Figure 4.4 An EEG signal channel is transformed into the basic wave 4.2 Hardware system The hardware system includes devices such as observation board, wheelchair model, Emotiv equipment and computers with control software, the hardware system is shown as Figure 4.5. Figure 4.5 Sitting posture and hardware devices 4.2.1 Observation board The observation board is 46x42 cm in size, can adjust the tilting direction like a laptop screen to suit each participants, on the observation board there are 5 images of 8x12 cm each placed at even intervals at a distance of 8 cm, a camera is placed between the image of the person and the flower to record the direction of the eye as Figure 4.6. Trang 25 Figure 4.6 Observation board Camera is mounted on the observation board to detect eye direction corresponding to 05 image types on the board, camera of Logitech Co. used with model C615, the specifications as follows: resolution HD 1080, 30fps, field of view 78o, connect to computer via USB, the camera is shown in Figure 4.7. Figure 4.7 Camera Logitech C615 4.2.2 Wheelchair model Wheelchairs used to simulate the process of commands from computer, the wheelchair with compact size can run forward, backward, right turn, left turn and stop in accordance with the commans form computer, the schematic diagram of the circuit is shown in Figure 4.9, specifications of wheelchair is shown in Table 4.1. Trang 26 Figure 4.8 Model of Wheelchair Table 4.1 Specifications of model wheelchairs No Dicriptions Specifications 1 size (length – width – high) 35x30x35 cm 2 Motor 200 rpm, 9 VDC 3 Number of motor 02 4 Battery 9VDC 2000 mA 5 Weight 0.8 kg Wheelchair control system principle schematic includes the arduino UNO3 processor, driver LM298 for two motors, and bluetooth module HC-05 to receive control commands from computer. Figure 4.9 Principle schematic of control board 4.2.3 Emotiv device Trang 27 An indispensable device in experimental control is the Emotiv EPOC + device (EPOC+ head), this device record EEG signals and send it to the computer via bluetooth, EPOC + device is shown in Figure 4.10. Figure 4.10 EPOC+ device EPOC + has the specifications as follows:  Number of channels: 14 channels + 2 reference channels.  Sampling frequency: 128 SPS / 256 SPS.  Data resolution 14 bits, LSB = 0.51 uV.  Connect by bluetooth, 2.4GHz band.  Battery using time 12 hours The computer connect to 2 devices, EPOC + head and wheelchair model as Figure 4.11. Figure 4.11 The computer connect to 2 devices via bluetooth 4.3 Selecting a group to participate in the system evaluation After finishing the above steps, author conducted the final experimental step to re-evaluate the entire research results. During this period, author selected 3 groups of participant as follows: Group 1: Selecting 20 from 60 students in the first phase of the best experimental results. Group 2: Choosing 20 from 60 remaining student. Group 3: Choosing 20 student never participated in the experiment test. Trang 28 In groups 1 and 2, student do not have to go through training steps, but group 3, student must be trained on the