Algorithm for alerting the unmanned aerial vehicle operator based on the image potential obstacle borders detection on the flight trajectory using the OPEN CV library

The article describes several modifications of the obstacle detection algorithm on the flight trajectory of an unmanned aerial vehicle (UAV). First, we give an illustration of the problems on the example of the relief image on topographic maps and the essence of the relief image by horizontal contours. Then the conclusion is made about necessity of development of algorithm and software, which can help the operator of the UAV in deciding on necessary trajectory changes of UAV, since, for example, guided solely by the method image of the terrain or another similar method in the planning of the UAV trajectory as preliminary preparation for the flight, however, such methods are fairly static and are not suitable in such situations as, for example, detection of unexpected obstacles. Further, the article describes the process of developing several modifications of the algorithm that solves the above problem and generates a notification of the UAV operator. It is also described depending on the flight speed of the UAV and natural conditions; it is possible to choose the most appropriate modification of the algorithm, which has different sensitivity to potential obstacles. The conclusion shows the settings screen and results of the software work and the principle of visual notification of the UAV operator with the proposal direction of the flight around detected obstacles.


Introduction
In today's world, received fairly widespread the unmanned aerial vehicles (UAV), which have a variety of purposes and often require using in conditions far from ideal.Thus, the problem of detecting obstacles on the flight trajectory, espe cially those that have arisen unexpectedly, since the design of UAV if bump with almost any kind of obstacle do not allow it to remain fully opera tional.
Therefore, it is necessary to design and imple ment an algorithm for detection potential obsta cles to the flight of the UAV, to develop software that uses streaming image acquisition from the camera.

The main part
To illustrate the problem, consider, for exam ple, the image of relief on topographic maps, which gives a complete and sufficiently detailed idea of the irregularities of the earth's surface, their shape and relative position, heights of the terrain points, the prevailing steepness and length of the slopes.Let's consider the essence of the re lief image by horizontal contours (figure 1).
Horizontal contour is a closed line represent ing a contour of irregularities on the map, all points of which are located on the ground at the same height above sea level.Contours can be rep resented as lines obtained as a result of the crosssec tion of the terrain by level surfaces, that is, surfac es parallel to the water level in the oceans [1].However, such images are quite static and are not suitable in such situations as, for example, the detection of an unexpectedly appeared obstacle, and as mentioned earlier, in the case of an un manned aerial vehicle, a bump with even small objects will almost certainly have catastrophic consequences for UAV.For this reason in this arti cle the most attention will be paid to the detection of dynamically appearing obstacles due to poten tial obstacle borders detection on the image and reaction of system after this.
Hence, the algorithm, step by step: 1) sensors reading (modeling); 2) get image from camera (in article using Open CV); 3) assignment of safe (green), potentially dan gerous (yellow) and dangerous (red) zones on the image for the purpose of detection of potentially dangerous objects and filtration of borders of the objects which are not dangerous (being aside from a trajectory of flight -in a green zone); 4) convert the obtained image into a grayscale (shades of gray); 5) detection potential obstacles on the image; 6) alerting the UAV operator (displaying of a message and playback of an audio signal) about the potential obstacle detection (subject to fulfil ment of previous step), indicating the most secure direction of the overflight of the obstacle (left or right and up or down) based on the detection of the area obstacle and its location relative to the UAV trajectory [2][3][4][5].
For greater clarity, we present the most signif icant steps of the described algorithm in the form of flowchart.Hence, step 2 of the algorithm (get information from the camera of an unmanned ae rial vehicle) is shown on figure 2.
As the device receiving the video stream, simu lating the UAV camera, were used Logitech cam era C110, with the following main characte ristics: number of megapixels: 0.3; resolution (photo): 640×480; resolution (video): 640×480; interpolated resolution (video): 1024×768; interpolated resolution in megapixels (pho to): 1.3 Mpix; maximum frequency: 30 Hz.Then, to research the different variants of sen sitivity to the borders of the image objects, con sider two variants of borders detection in images (steps from the 3 rd to the 5 th of algorithm), prog rammed in the unsafe context of the C# prog ram ming language, to accelerate the image processing.
1) method of the image binarization by thre shold (figure 3, where illustrate the preparation procedures for analyzing the original image and figure 6, where illustrate the obstacle detection procedure) as the most sensitive to detection the obstacles of those considered, therefore it can be used only in conditions close to ideal [6][7][8].
Note: since memory is a linear structure, the entire image will be represented in computer me mory not like a matrix, and not like we imagine the image in a life (each pixel consists of three colors: red, green and blue), but the entire image will be represented as a linear structure, where each pixel is represented as three values (without transparency -an additional alpha channel) -fig ure 4.
It should also be noted that the step 4 is ob tained by applying the known color ratio coeffi cients of the RGB model of the main image) and The 3 rd step is the division of the image into logical sectors-zones (the size of the sectors selected empirically and, for example, the red zone is equal to the fourth part of the image, because this size is sufficient for the flight of the UAV between the objects placed in the yellow zone on the left and right or up and down, if the analysis is conducted from a distance of approximately 5 x b, where: b -the width of the UAV, with the characteristics of the camera similar to those presented earlier) [9][10][11][12][13][14][15].This algorithm step most fully reflects the essence of the development of our algorithm as a whole, which consists in separating the processing logic of potentially dangerous zones based on calculating the areas of obstacles in these zones with automated camera calibration and taking into account the flight speed of the UAV, wi thout the need for additional measurement equipment.
The process of camera calibration and zones visual representation is shown on figure 5.
A breakdown into zones flowchart is presented below (figure 6):   2) the Kirsch method.To prevent of duplica tion elements in flowcharts, will be presented only the key features, in comparison with the method of binarization by threshold value [15].The Kirsch method has an average level of sensitivity to ob stacles.Algorithm fragments are shown on fig ures 7.
The last step of the algorithm is to alert (using developed software) the operator of the UAV which controls the flight about potentially dan gerous situation and possible bump (if an obstacle was detected) with the displaying of an alert mes sage and playback of an audio signal to attract the attention of the operator.
The operator is also informing about the most secure direction of the overflight of the obstacle, according to the area of the obstacle in the yellow and red zones (right (left) or up (down)).
The developed software has two tabs: «Set tings» and «Simulation», three control buttons «Start» -to run the simulation based on the set tings, «Stop» -to stop the modeling process and «Save log» -to generate a report involves the set ting values and other working events of software to a text file [16][17][18].
The «Settings» tab is shown on figure 8: According the figure above the «Settings» tab involves the next settings: threshold of sensitivity to the increase of speed -the greater the speed, the more algorithm sensitivity to obstacles, because with speed increas ing the maneuvering should do as soon as pos sible; data sampling rate -algorithm trigger fre quency in seconds; method for detecting borders on the imagea dropdown list that allows you to select the de sired method with different sensitivity, because the choice of method should vary depending on the experiment conditions.Possible options: Bi narization by threshold, Kirsch method, Laplace method, Roberts method, Sobel method, Wallace method and auto selection.
Tab «Simulation» is divided into four logical zones: top left -display streaming video from the camera (original image, before the analysis); top right -display the result of the analysis with a visual representation of zones described in the third step of the algorithm; alerting zone (for display alert message du ring detection of a potential obstacle); part of the software log (for display excep tions of the hardware component working or fai lures in the software itself).
Alert situation where was obstacle detected is shown on figure 9: Flight in a normal mode is shown on figure 10: Figure 9 shows a result of execution the algo rithm, where the borders of the potential obstacles detect by using of Laplace method.All options are shown in figure 8 (dropdown list with choos ing of method for detecting borders on the image) As shown on the figure, if the UAV was directed to an object that potentially threatens it with a bump, the operator receives a visual alert (togeth er with an audio signal) with indicating the direc tion in which it is proposal to direct the UAV.
Figure 10 shows the situation of simulation the flight of UAV in nonideal conditions (in con ditions of high clouds of the autumn sky) in order to determine the number of false alerts and estab lish the algorithm's insensitivity factor.
It should be noted the most important stage of the software execution is using of an obstacle de tection algorithm based on the programmed pro cessing logic of the zones potentially dangerous for flight (for convenience, they are highlighted in color and are called respectively), and the opera tor notification subsystem is easily replaceable and is intended to demonstrate the algorithm exe cution.In the future, due to the low cohesion and modularity of the developed solutions, the opera tor alert subsystem can be easily replaced with the sending control impacts subsystem to the UAV.

Conclusion
In this paper, the image of relief on topogra phic maps was raised.Than the conclusion was made about the need to detect and respond to ob jects around UAV, the presence of which, nothing was known in advance.The algorithm for detection of potential ob stacles to the flight of UAV was designed and im plemented.Next, software that get video stream from the camera using the Open CV library, that used above mentioned algorithm in several modi fications, and provides visual and audio alert for the user (UAV operator) about need to reaction to the potential dangerous situation.Also, software can propose of a possible direction of flight around the obstacle, where, according to the soft ware, less chance of a bumping with a potential obstacle.
As a further development of this theme, it can be proposed to take into account the distance to the potential obstacle according the flight speed of the UAV, to be able to estimate the time range available to the operator for decisionmaking about control of the UAV.Email: evgeni.hvitko@bntu.by

Figure 1 .
Figure 1.Image of relief by horizontal contours on topographic maps

Figure 2 .
Figure 2. Receiving video stream from Webcamera with image conversion using Open CV

Figure 6 .
Figure 6.A breakdown into zones flowchart

Figure 7 .
Figure 7.The key features of Kirsch method in comparison with the method of binarization by threshold

Figure 9 .Figure 10 .
Figure 9. Alert situation where was obstacle detected Stepanov V. Y. Received the Software Engineer and M.S.C. degree in field of «Information systems and technologies» from the Belorussian National Technical University in 2013 and 2014 respectively.He is currently working toward in PhD degree in the field of «Digital signal processing and control» including, paramet ric optimization and programming.His research interests include IT, program ming and robust control.Email: vladimir.stepanov@bntu.byHvitko E. A. Master Student of the department «Information Systems and Technologies» of the International Institute of Distance Education BNTU.Lectur er of the Department «Software Computing and Automated Systems» of the fac ulty of «Information Technologies and Robotics».His current research interests include control of unmanned aerial objects.