1、1Robot visual design with the panorama synthesis function（2.School of Information Science and Technology , Tsinghua University, Beijing 100084,China） Abstract. Panorama synthesis is basis of the target tracking. Through the synthesis of panorama, can enlarge the visual range to capture for the camer
2、as, in the big background, the suspicious target which needs to be tracked are more easily detected, then selects the target, and Tracks it. In the existing target tracking of robot integrated machine, adds panorama synthesis function and the suspicious target selection function, the target tracking
3、 runs only on selected target, make the tracking more targeted. In the Panorama synthesis module of the present system, there are two ways to obtain video stream, that is real-time image stream from Cameras and the local AVI video file. Through the algorithm to judge the position relationship betwee
4、n two adjacent frames, finishes the image stitching, and displays panorama of the in the user interface in real time. By a particular target selection from users, complete moving object detection and tracking in motion 2camera environment. After the test, the system can meet the general needs of app
5、lication oriented robot visual. Key words: panorama; synthetic algorithm; robot integrated machine; target tracking; interaction 1.Introduction In some fields, wide view and high resolution images or video is becoming more and more important, Such as structure of panoramic view in photographic surve
6、ying, background reconstruction technique in the field of video coding, Realization of panoramic video monitoring system and virtual environment construction in virtual display technology. In order to obtain a wide view scene image, people must by adjusting the camera focal length or the use of spec
7、ial equipment (Panorama wide-angle or fish-eye lens) to uptake in intact scene, but it acquired relatively low resolution panoramic photo, therefore, people used image stitching technology to splice More photos into a large panorama. Along with the scene expansion, the image sequence number used for
8、 splicing increased, Processing time generated a panorama is longer. Therefore, In the premise of quality assurance to generate images, to improve the panorama generation rate and to achieve the accurate real-time even real-time requirements are 3the hotspot and difficulty in the present research 1.
9、 According to different projection planes, panorama can be divided into cylindrical, spherical panorama, cubic panorama. Cylindrical panoramic image technology is now mature, but it is only suitable for image sequence stitching obtained from the cameras one-dimensional rotation along the horizontal
10、direction; cubic panorama research is deepening, but effect is not as good as spherical true; spherical panorama is suitable to describe the large range scene, accord with peoples habits of observation, but the model is complex, image sequence acquisition is difficult, related research is still imma
11、ture 2. Image registration is the key in Panorama generation process, the most commonly used methods have method based on motion3, method based on characteristic point 4, method based on manifold projection 5 ,etc, these methods have the large amount of calculation, Panorama generation rate is slowe
12、r, they are limited in practical application. On the other hand，in the visual monitoring system, in order to detect suspicious cases, need to segment the moving objects in the scene, obtain objects described, and then to pay attention to it suspicious whether or not. In some special scenes, also nee
13、d to judge distance from the suspicious objects 4to a particular, such as not too close from some restricted zone, or need to make a warning. In many applications, there are some common features，one is required to detect some objects in the video stream, the other one is analysis and tracking of vid
14、eo information, and selectively stores. People often only care about the video content when the accident occurred, whereas in the past various visual system , all the video information are preserved, which leads to a tremendous data redundancy, not only occupies a large space, but also decreases the
15、 efficiency sharply to manually find abnormal events. And if you can understand the video information, only save the parts to pay attention to, the situation will be improved greatly. In addition, in these scenes are often necessary to let the camera movement, as is often needed to monitor a larger
16、region, in only one camera case, only to let the camera continuously scan the entire scene, and finish panorama synthesis. And in robot system, camera is usually erected on the robot, so it must move following robot motion at the same time. Visibly, in Motion camera environment, the segmentation, th
17、e recognition and tracking of the foreground object are very meaningful. Only the middle-level information is obtained, then high-level event reasoning begins 5possibly. This research work was based on the above two aspects - a panorama composition, to extend the camera saccadic range; target tracki
18、ng, keep the target in the lens and has been captured. 2. System structure and scheme constraints 2.1 System structure The robot vision mentioned in the system refers to camera mounted in application oriented Robot Integrated Machine (RIM) remote control scheme, the scheme structure diagram as shown
19、 in Figure 16. Fig. 1 RIM remote control scheme system structure diagram In Figure 1, the camera is an integrated machines visual hardware, it is responsible for the entire field monitoring. According to camera information and feedback information from teleoperation platform, the operator can accura
20、tely grasp actual situation of RIM, and control movement of RIM and camera. In which, realized application oriented dynamic target tracking. This article is based on this, to give RIM with a panorama composition function and suspicious target tracking function, so that robot visual function is stron
21、ger and more practical. Panorama synthesis process and the target tracking process can 6switch each other. 2.2 scheme constraints As first step of the visual system design, a panorama composition is the foundation of target tracking. Through panorama composition, can expand visual range captured by
22、a camera, in the large field of vision, the target needed to track are more easily detected. The panorama synthesis module realized in the system gets the video stream in two ways, the two ways are to capture real-time image stream by the camera and AVI video file stored in the local. Through the al
23、gorithm, the system judges the position relationship between two adjacent frames, finishes the image mosaic, and real-time displays the current state of panorama on the user interface. The system agrees: after RIM movement, camera is fixed to the horizontal scanning direction (without considering th
24、e pitch angle), and thus greatly simplifies the calculation of panoramic image mosaic. Upper and lower boundaries of two adjacent frames are aligned in the horizontal direction, only needs to find matching points and to splice. Of course, in the splicing process, the system real-time judges whether
25、camera saccade direction changes. 7Thus, the design goal of panorama synthesis module is to use the efficient and accurate matching algorithm, to Synthetic image in real time, to judge camera saccade direction in real time, in order to determine whether needs to update part of panoramic image which
26、has been completed stitching. Eventually the panorama is displayed on interactive interface in real time. 3. A panorama composition design 3.1 Algorithm design A panorama composition module is one of the two basic function modules in the system, the module function is to complete panorama synthesis
27、of the video file, to broaden the field of view of the camera. The system designed two input methods for the module, they are to capture video from a camera directly and read the local video file. If you choose mode to capture a camera video, but also video file can be written to the local hard disk
28、. Similarly, panorama image files can be written to the local hard disk. In the synthesis of panoramic image, need to display part of the synthesis of real-time image to the user, and need to judge shooting direction of camera or shooting direction of video files from the video, to decide whether ne
29、ed to begin to synthesize another panorama 8file 7. The system design flow chart of a panorama composition module is as shown in figure 2. In this module operation, several states require to be judged in real-time, namely when video frames need to be judged, they are judged: firstly, to judge whethe
30、r a system state is changed, the user can switch states of the system in a panorama composition process, for example, to convert to target tracking function, without waiting for the end of reading video; secondly, to judge whether reading video is end or not, this is mainly designed for input mode o
31、f local video file, for input mode capturing from the real-time camera , you can ignore this judgment; the last is the direction judgment, after reading a new frame every time, comparison is needed between the previous frame and the last frame, Purpose is to judge the shooting direction of video str
32、eam, if the video direction changes, existing panorama synthesis results is needed the corresponding treatment according to user requirements, and open up storage area of new results, otherwise according to the direction, part of a new frame which does not exist in synthetic maps is stored to storag
33、e area of results. 3.2 Module Realization This module uses the C+ language as a programming 9foundation, and combined OpenCV. OpenCV is an open source computer vision library, which is a series of C function and a small amount of C+ 8, it realized many common algorithms the image processing and comp
34、uter vision aspects. In this paper, template matching algorithm used in a panorama composition process is simplification of spherical projection model, mainly used the gray-scale image matching algorithm. Matching algorithm based on the gray projection 9 is that two-dimensional image gray values are
35、 projected respectively and transformed into two independent groups of data, then On the basis of one-dimensional data, begins image matching. As a result of dimension reduction, greatly reduces the calculation amount in the matching operation, thus greatly improves the matching speed. At each step
36、of the operation, first of all operates the image space transformation of two images ,change RGB space into the gray space, reduce the dimensions of the image, to improve the matching speed. And then through the template matching algorithm, judge Position relation between the current frame and the p
37、revious frame, thus determines the direction of video, then finishes the image synthesis. OpenCV provides powerful algorithms library, here is several major functions used by this module 10. 3.2.1 10CvtColor-Color space conversion function The most commonly application of RGB (red, green, blue) is t
38、he monitor system, it can display each color value for HSB, RGB, LAB and CMYK color space. HSV (hue, saturation, value ) color space model corresponds to a subset of conical in the cylindrical coordinates in, Cone top surface corresponds to V=1. that is to say, the V axis in HSV model corresponds to
39、 the main diagonal in the RGB color space. The color on the cone top surface circumference, V=1, S=1, this color is pure. HSV model corresponds to the painter matching method. Artists obtain different colors from some color by the method of changing the color consistency and color depth, adds white
40、in a solid color to change color consistency, adds black to change the color depth, while adds different proportion of white and black at the same time, artists can obtain different colors. HSI color space is from the human visual system, it describes the color using the tone (Hue), color saturation ( Saturation or Chroma ) and luminance ( Intensity or Brightness ). Therefore, in the HSI color space can greatly simplify workload of the image analysis and processing. HSI and RGB color space is different notation of the same physical quantity,
