Tóm tắt Luận án A research into the control of mobile robot manipulator to track target based on visual information

, Thai Nguyen, Dec 12-14, pp. 217 – 227. [5] Le Van Chung, Pham Thuong Cat (2014), “Robust visual tracking control of pan tilt – stereo camera system g i pp.167-173. [6] hu g h h g t i u h 013), “Phương pháp điều khiển hệ servo thị giác stereo sử dụng bệ Pan-Tilt - - g - 382. [7] hu g “Phát triển hệ pan/tilt – nhiều camera bám mục tiêu di động”, Thai Nguyen University Journal of Science and Technology, 116(02), tr. 41-46. 1 INTRODUCTION In recent years, there has been a great deal of research on robotics control using visual information. But the achieved results still reveal some limitations. For example, using a camera on a mobile robot only allows full tracking of the target's information when knowing the target's moving plane. As with the use of two cameras on the pan tilt system, but not considering the deterioration of the Jacobian matrix affecting the grip capacity of the system. Besides, the mathematical model of a robot is often difficult to achieve the required accuracy because there are many unspecified parameters in the system such as measurement of parameters or coefficient of friction, inertia mo e t etc, usually changes during operation. In addition, it is difficult to optimize the parameters in the robot controllers to achieve the desired accuracy. With the above reasons, the author has chosen the topic: “A research into the control of mobile robot manipulator to track target based on visual information to develop some control algorithms that use image information with many uncertain parameters. The objective of the thesis The main object of the research is to focus on pan-tilt robots and mobile robots with wheels. Scope of research Researching methods of controlling pan-tilt tracking moving target using image information from 2 cameras with many uncertain parameters. Developing algorithms for controlling integrated system include mobile robot, pan-tilt and two cameras tracking moving targets. New findings of the doctoral dissertation 1- The application of artificial neural networks in compensate for uncertainties in a pan-tilt system model with two cameras. Based on that, the kinematic and dynamic controllers are constructed for the pan-tilt two cameras system to track moving targets with uncertain parameters. 2- The thesis has developed a dynamic model for the integrated system that including mobile robot, pan-tilt and stereo cameras. Also, 2 the thesis has built two control methods, sliding mode controller and quadratic perfofmance optimal controller for the above integrated system. The layout of the thesis Chapter 1: Overview. Chapter 2: Developing controller for pan-tilt stereo camera system to track the moving target. Chapter 3: Some improvements in controlling servo system to track the target. Chapter 4: Developing control method for mobile robot. CHAPTER 1 OVERVIEW 1.1 Problem? In order to control a robot system using two cameras to work better, the problem is: The first: Developing methods to control pan-tilt systems using image information to track moving targets when there are uncertain parameters. The second: Build a Jacobian matrix image is a square matrix for the system to track the moving target for the better system. The third: Developing some methods to control the integrated system combining a mobile robot with a pan-tilt robot that carries two cameras to track the moving target and be able to move closer to the target in the space. 1.2. Overview of controlling robots using visual information When using a pan-tilt robot with two cameras, another problem poses the degradation of the Jacobi matrix when taking inverse pseudocode. When using a camera, the Jacobian matrix of the pan-tilt system - a camera is square and invertible. But when using 2 cameras, the Jacobian matrix of the system will be (3x6), the Jacobian matrix of the system will be (3x2). So in transformations we have to take the inverse pseudo cause that is the cause of singularities. 3 Hình 1.2 Some methods of controlling robots 1.3 The research issues of the thesis - Building kinematic/dynamic model of pan-tilt robot and developing classical control method combining with Neural Network to get better results including: - Research and improve to build a square image Jacobian matrix - Developing advanced control methods for robots when having uncertain parameters. - Building dynamic models for the integrated system that including mobile robots, pan-tilt with 2 cameras and controllers for the above system. - Using Lyapunov stability method and Barbalat's lemma proves stability and Matlab to verify results. Chapter 2 DEVELOPING CONTROLLER FOR PAN TILT STEREO CAMERA TO TRACK MOVING TARGET In this chapter, the kinematic control algorithm combined with the neural network is built to control the rotation angle of the pan - tilt robot so that the target image is always maintained at the desired Optimal paramet ers Optimal output Control robots using visual information Classical method Kine- matic Dynam -ic Modern method Sliding mode Combined with neural network Optimal control, adaptable Modern methods with neuron network Classical methods with neuron network 4 position on the model of the image frames. The content of the chapter consists of 4 main parts: building a kinematic model of the pan-tilt stereo camera system, designing control algorithm, verifying and comparing with the controller not associated with Neural network and conclusions. 2.1. Kinematic model of stereo visual servoing system with uncertain parameters 2.1.1 Determination of Image Jacobian matrix: Fig 2. 1 Pan-tilt PTU-D48E-Series & camera coordinates The velocity of t rget’s i ge o c er s: ( )imagm = J m v (2.9) - is the image Jacobi matrix: - v is the velocity vector of the camera system. 2 2 ( ) 2 0 2 2 2 ( ) ( ) 0 2 2 2 2 ( ) 2 0 2 imag U U U U U U V f U U UR L L L R L L L L R VL B fB f f U U V U U f V V U U U UR L L L R L L L R L R B fB f f f U U U U U U V f U U UR L R L R R L R L R VL B fB f f J (m) (2.10) 2.1.2 Kinematic equations of Pan-Tilt platform: Denote robotJ the Jacobian of the Pan-Tilt platform, we have: (2.11) imagJ (m) ( )robotx = J q q 5 Fig 2. 2 Camera system model 2.1.3 Formulation of stereo visual servoing problem with uncertain parameters: We calculate the image feature error: dε m m The kinematic control problem of stereo visual servoing is to find control law ( )q = K ε to control the system track the moving target so the tracking error ε converges to zero imagm = J (m)u ; robotv = x = J (q)q ; q = K(ε) 2.2. Control law design In this chapter, a neural control method is proposed for Pan - Tilt - stereo camera system to track a moving object when there are many uncertainties in the parameters of both camera and Pan-Tilt platform. ˆ , d d 1ε =m-m = Jq+f -m -Kε+f +u (2.21) The structure of the chosen artificial neural network is of RBF type as shown in Fig.6. It has three layers: the input layer includes the three components of error ε , the output layer includes 3 linear neurons and hidden layer contains neurons with Gaussian output function: ; j = 1,2,3 (2.22) 2 2 exp j j j j c 6 Fig 2. 6 Structure of proposed visual tracking system with many uncertain parameters. Control Method 1: The stereo camera system described by the model in Esq. (2.9), (2.11), with uncertain parameters controlled by the neural network defined by Eqs (2.22), (2.23) will track moving targets with the error , ε ε 0 if the speed of the Pan-Tilt joints is determined by the Eqs. (2.25), (2.26), (2.27) and learning rules (2.28): ˆ ˆ ˆ+ , + + , d 1 d 1q J [(m - Kε)+u ] J (m -Kε)+J u 0 1= u +u (2.25) 0 ˆ + du = J m -Kε (2.26) 1 ˆ , 1 + , 1 1 ε u = ( )Wσ - ε u = J u (2.27) T W εσ (2.28) where K is a positive definite symmetric matrix. TK = K > 0 , the coefficients 1, 0   . Proof: Chose a positive definite candidate Lyapunov function as follows: 3 1 1 2 i i i V T Tε ε w w (2.29) 0V Tε Kε ε (2.37) 2.3 SIMULATION Simulation 1: Fixed target Camera center at the initial time: m(0) = [-40, 30, 0] (pixel); Image coordinates of the target stand still at mt= [-20, 0, 20] (pixel); 7 Fig 2. 7 Image feature error when using neural control , 1u 0 Fig 2. 8 Image feature error when no neural control is used , 1u = 0 Simulation 2: Moving target in a straight line Moving target from point A(0m,1.8m,0m) to B(-0.3m, 1.8m, - 0.5m) on the plane ZCOCXC Fig 2. 9 Tracking error coordinates when the target moves along a straight line. Fig 2. 10 Image feature error coordinates when the target moves along a straight line. 8 Fig 2. 11 Tracking error coordinates when the target moves along a straight line and no neural control is used. Fig 2. 12 Image feature error coordinates when the target moves along a straight line and no neural control is used. Simulation 3: Moving target in an arc The target follows the circular arc with center coordinates at the origin O(0, 0), radius r = 1 on the plane ZCOCXC Fig 2. 13 Tracking error coordinates when the target moves along an arc. Fig 2. 14 Image feature error coordinates when the target moves along an arc.. Simulation 4: Moving target in arcs with acceleration. 9 The target follows the circular arc similar to the arc of simulation 3. However, in the first 1/6 arc, the velocity increases steadily with acceleration 1cm/s 2 , after 3 seconds it will move with the constant speed 3cm/s. In 1/6 end of the arc, its moving speed reduces with the deceleration -1cm/s 2 Fig 2. 15 Tracking error coordinates when the target moves along an arc with changing Fig 2. 16 Image feature error coordinates when the target moves along an arc with changing 2.4. Conclusion of chapter 2 This chapter proposes a new control method for the stereo visual servoing system using neural networks with on-line learning rules to compensate for the impacts of uncertain parameters such as inertial torque, Jacobian matrix, friction in the joints, noise effects, etc he ro osed ethod gu r tees the st bility of the over ll system and eliminates tracking error. Control algorithm is highly adaptable and is able to resist the noise impact on the system. The global asymptotic stability of the whole system is proved by the 10 Lyapunov stability theory. The simulation results with the uncertainties up to 20% in the case of fixed target, moving target in a straight line or circular arc, show that the tracking error converges to zero. These results are consistent with the theoretical. . CHAPTER 3 SOME IMPROVEMENTS IN CONTROLLING SERVO SYSTEM TO TRACK TARGET As in chapter 2, the author has noticed that in order to control the pan- tilt system with two cameras to track target work well, the control problem has some issues to solve as follows: - Firstly, it is necessary to build a square image Jacobian matrix so that performing the inverse and avoid singular points leading to losing grip. - The second is building dynamic controllers in combination with neural networks to compensate for the effects of uncertainty parameters inside the model as well as external noise. - The third is to optimize some parameters in the neural network to get better outputs. In Part 1 of this chapter, the author built a 3D model for two camera system to obtain the full Jacobian matrix. In part 2, the author built the dynamic controller using neural network with optimized parameters, the stability of the system is demonstrated by Lyapunov method and Barballat lemma. Part 3 is the simulation results. Finally, some conclusions. 3.1 3D visual model for eye-in-hand stereo camera system 3.1.2 3D virtual stereo camera model systems A 3D visual space is built according to the following steps: First step, from geometrical relations between the target and that feature images we calculated the coordinates of the target point  Tc zyxx in the camera coordinate frame cccc ZYXO Second step, a reference coordinate frame with the origin located at the same position as is defined. In order to transform OC to OV, the rotation matrix (Fig 3.2) is used. The vvvv ZYXO cccc ZYXO v CR 11 Fig 3.2 3D visual stereo camera model projection of  Tc zyxx in is defined in as  Tvvvv yxzx Third step, the reference coordinate frame vvvv ZYXO is used to define two virtual camera's frame 1111 vvvv ZYXO , 2222 vvvv ZYXO associated with stereo cameras. Their location on Xv and Zv axes are far away from Ov the distances λ Last step, the virtual camera model is combined with 3D visual camera model to construct a 3D virtual Cartesian space having feature point vector denoted as:  Tvvvs xzz 121x . vimgJ is the visual Jacobian matrix: vvimgvs f xJx   (3.9)                              vv v vv v v v v vimg zz y zz x x z x 1 0 )( 0 1 )( 0 )( 1 2 2 2 J (3.10) 3.1.3 Avoid singularity cccc ZYXO vvvv ZYXO 12 The singularity of vimgJ that can be avoided by choosing the r eter λ such s λ > x xv, zv). 3.1.4 Stereo visual servoing problem with uncertain parameters 0) ˆ( xJqJJx C v vimgvs Rf   . (3.19) 3.2. Dynamics of robot manipulator with uncertainties. The dynamics of a serial n-link rigid robot with friction, and uncertainty can be written as follows: τdqgqqqCqqM  )()(),()( t (3.20) 3.3. Robust neural control of stereo camera system with pan tilt robot. 3.3.1 Construction of robust controller The control law is chosen as follows: 0 1( )D P NR      τ A K ε K ε b τ τ τ , (3.30) 0 ( )D P   τ A K ε K ε b , (3.31) 3.3.2 Layer construction of RBF neural network The structure of choosing artificial neural network is a Radial Basis Function (RBFNN) network. It has 3 layers. Input layer is vecto  Tsss 321s Hidden layer computation The hidden layer consists of neurons with output function calculated by Gaussian form. Output layer The output values of the network are approximate function f1 Control Method 2: The image error dynamics (3.32b) of the uncertain pan-tilt – Stereo camera tracking system (3.19), (3.20) will be asymptotically stable with the error if the control torque is chosen by following (3.36), (3.37) and online learning rules (3.38): (3.36) (3.37) , (3.38) 0ε ( )D P NR    τ A K ε K ε b τ  1NR           ε τ A Wσ ε T W sσ 13 Fig 3.6 Structure of proposed visual tracking system Proof: We choose the candidate Lyapunov function as follows:    3 1 2 1 2 1 ),( i i T i TV wwssWs . (3.39) Taking the derivative of V along time, yields: 0 sGss TV (3.49) 2 T TV                     s s s G β Wσ β Gs s s (3.50) We found that V is bounded because βs, are bounded, s s is the unit vector, G is the positive-definite constant matrix and 0,  . Thus V is uniformly continuous. According to Barbalat's lemma, we have 0s  when 0t and it forces 0ε and the system is asymptotically stable 4. Simulation results Simulation 1: Moving target in a straight line from point A (0m,3m,0m) to B (-0.5m, 3m, -0.3m) 14 Fig 3.7. Tracking error coordinates when moving target in a straight line Simulation 2: Moving target in a circle O (0, 0, 0), r = 1 Fig 3.9. Tracking error coordinates when moving target follows a circle. 15 a) b) c) Fig 3.11.a) Tracking error coordinates in X, Z axes. b) Torques of pan and tilt joints. c) Joints angle q Simulation 3: Moving target in a rectangle with changing velocity. Moving target follows the rectangle as figure 8 from point (-1m, 3m, 0m) to (-1m, 3m, 1m) in 10 seconds. In the next 10 seconds, target moves from point (-1m, 3m, 1m) to (1m, 3m, 1m) and the same to return to the starting point. Fig 3.12 Tracking error coordinates when moving target is in a rectangle with changing velocity. 16 a) b) c) Fig 3.13 a) Tracking error coordinates in X, Z axes. b) Torques of pan and tilt joints. c) Joints angle q Simulation 4: Moving target in space with random velocity and direction. The start of target at point (3, 3, 0.5) in O0 coordinates; yt = 0.1t. The movement of the target following x, z axes is plane-parallel motion: 0 ≤ t <5s: v = /s) ω = r d/s) 5 ≤ t <10s: v = 0.5 + 0.15sin((t- )π/ ); ω = 0.15sin((t-5)π/10); 10 ≤ t <15s: v = 0.75. ω = 0; 15 ≤ t <20s: v = 0.75+0.15sin((t-15)π/10); ω= -0.15sin((t-15) π/10); 20 ≤ t <25s: v = 0.75. ω = -0.15; 25 ≤ t <30s: v = 0.75. ω =-0.15 - 0.15sin((t-25)π/10); t >30s: v = 0.5, ω = -0.3. λ = 20. Fig 3.14 Tracking error coordinates when moving target is in space with random velocity and direction. 17 a) b) c) Fig 3.15 a) Tracking error coordinates in X, Z axes. b) Torques of pan and tilt joints. c) Joints angle q In the simulation 1 - moving target in a straight line in Fig. 6, it is found out that the error converges to zero. When target follows a circle (simulation 2), the system can track targets, but the error is still great so it needs to improve. When the function cos or sin is changes from positive to negative and in the opposite at ¼, ¾, 0, ½ circular of the circle, the tracking target error is fluctuations, but it still tracks the target. Different from chapter 1, it lost tracking at ½ circular. In simulation 3, 4, when the target moves following a rectangle and random in the space, the system still captures. When the moving target simultaneous changed orientation and velocity, the initial error of the system increases, but it still tracks target rapidly because external effects as well as the effects of uncertain parameters in the system model are well compensated by the neural network controller based on-line learning algorithm. The impact of noise and uncertain parameters is reduced. Otherwise, image and robot Jacobian matrices (Jimg and Jrobot) have not a singularity. This condition helps the system move follow in the complex way. Fig 3.17. Tracking error coordinates without part of neural network in the controller when target moves following a straight line and a circle 3.5. Conclusion 18 The problem of visual tracking control of pan-tilt-stereo camera system is dialled with when there are uncertainties in both the Jacobian matrix and the dynamics of the system. A 3D virtual stereo camera model having a full-rank of image Jacobian is constructed to solve local minima and image space singularity problems of the classical image Jacobian. Furthermore, the authors propose a robust visual control scheme with online learning RBF neural network to compensate the effects of uncertainties. Due to online learning algorithms of RBF neural network continuously update is done only with some multiplication and integral. Therefore the calculations should not be too large, in accordance with real-time system. The asymptotic stability of the overall system is proved by Lyapunov stability method. Simulations show that the proposed control scheme is relatively effective even in the case of the target moving in a circle or rectangle meanwhile the dynamics and the Jacobian of the pan-tilt robot are not known exactly. In the other hand, all parameters in the controller can be optimized to have better results. In this paper, I only optimized two parameters. Other parameters such as ƞ, δ, G, H also can be optimized. I will consider all effect of the optimization parameters in the next step of my research. Chapter 4 DEVELOPING CONTROL METHOD FOR MOBILE ROBOT In this chapter, the system is adjusted so that the target image is always maintained at the center of the camera coordinate system. The wheels are put in the right forces to move and approach the target with time and energy consumption is optimal. The chapter content consists of 4 main parts: building the dynamic model, dynamic of the mobile robot – pan tilt - stereo camera system, designing a controller to control the integrated system to approach the target. In the shortest period of time, simulating verification and conclusions. In this chapter, the author presents the optimal control algorithm and sliding mode control algorithm combined with the direct moment to control the whole system. 19 Fig 4.1 Structure of mobile robot- pan/tilt-stereo camera 4.1. Construct the kinematic model and kinematic controller for mobile robot - pan tilt - stereo camera to track a moving target 4.1.1 Determination of Image Jacobian matrix The images Jacobian is defined follows the equal (2.10). 4.1.2 Kinematic controller for pan-tilt robot. The desired velocity of pan - tilt joints is: as 1 as1 1 as 2 2 ( ) l t l td pl t d m J Km m m (4.13) The desired coordinates, directions of the mobile robot is calculated by: 0 1 0 1 0 0 cos sin atan 2( , ) md t b md t b md b t t x x k y y k y x           (4.16) 4.1.3 Determination of the speed of the mobile robot wheels to reach to the target. The desired angular velocity of the two wheels of the mobile robot is: 1 / 4 / 4 / 2 / 2 rd m ld m vd d d k d k                   (4.21) 20 4.2. CONTROLLER FOR DYNAMIC SYSTEM 4.2.1 Dynamic model for mobile robot – pan/tilt system The dynamics of integrated mobile robot – pan - tilt robot with no friction or effect of noise can be written as follows: ( ) ( , ) ( )s s sM q v C q v v g q = τ (4.22) The tracking error e1 is: 1 d se v v (4.25) 4.2.2 Optimal Controller Design We choose control law as follows: * 1τ h He u (4.30) * 1 1 2 ( , ) ( ) T T F tt s u R B R e s (4.43) Control method 3: Optimal control signal u* (4.43), control law (4.30) will make the dynamic system in the equation (4.22) tracking the target with desired velocity and the objective function (4.33) minimum. If the parameters satisfy the condition (4.39, 4.40, 4.41). Fig 4.5 Structure of system 4.2.3 Design Adaptive CTC controller for dynamic system. The adaptive control law is designed as: 1 1ˆˆ , , ( )T T s a Γ D q q v M q B Ps (4.49) Where Γ is r × r) di go l ositive - definite constant matrix, and P is a (2n ×2n) symmetric positive - definite constant matrix satisfying T A P PA Q , (4.50) Q is a symmetric positive - definite constant matrix. 21 Fig 4.6 Structure of system 4.3. SIMULATION 4.3.1 Simulation with optimal controller Mục tiêu di chuyển trong không gian theo quỹ đ o h s u: xu t phát t i vị trí (x, y, z) = (2, 4, 0) so với gốc tọ độ s u đó tọ độ tọ độ z = 0.1t. Còn tọ độ x y thì th y đổi theo vận tốc thẳng và góc h s u: t = 0-5s: v=0.5m/s. ω = 0 rad/s. t = 5-10s: v = 0.5 + 0.15*sin((t-)*pi/10); ω=0.15*sin((t-5)*pi/10); t = 10-15s: v=0.75m/s. ω = 0 rad/s. t = 15-20s: v = 0.75 + 0.15*sin((t-15)*pi/10); ω =- 0.15*sin((t-15)*pi/10). t = 20-25s: v=0.75m/s. ω = -0.15 rad/s. t = 25- 30s: v=0.75m/s. ω =-0.15 - 0.15*sin((t-25)*pi/10); t >30s v=0.5m/s. ω = -0.3 rad/s; Fig 4.8 Tracking target of stereo camera. Fig 4.9 Tracking target of mobile robot 22 Fig 4.10 Error e between desired and measured angular velocity 4.3.2 Simulation with Adaptive CTC controller The target moves following an arc O(5.5m, 0.5m), r = 4m. in 20s. Fig 4.11 Tracking target of mobile robot Fig 4.12 Error e between desired and measured angular velocity 4.4. Conclusion chapter 4 In this chapter the kinematic and dynamic models of the integrated system are constructed. In addition, the authors propose a quadratic performance of optimal control for the system. The pseudo-inverse matrix of Jp is reversible at -π/ circle of the target movement, but it still causes of shock at about 35 seconds (Fig 4.9). However, we will research about inverse properties of it in others paper. The research results are presented in document [3] & [4] of the list of 23 published works. 5. CONCLUSION 5.1. The main research content of the thesis The thesis has implemented the following main contents: - Analyze control methods for pan-tilt robot to develop new control methods. The thesis has proposed 1 kinetic control method and 01 dynamic controller using artificial neural network for pan-tilt system using 2 cameras to track a moving target. The control methods are well proven by Lyapunov stability the