Fast-Direct Optical Collision Avoidance System Based on FLIR Image Sequences
In this white paper, a forward-looking infrared (FLIR) video surveillance system is presented for collision avoidance of moving ships to bridge piers. An image pre-processing algorithm is proposed to reduce clutter background by multi-scale fractal analysis, in which the blanket method is used for fractal feature computation. Then, the moving ship detection algorithm is developed from image differentials of the fractal feature in the region of surveillance between regularly interval frames. When the moving ships are detected in region of surveillance, the device for safety alert is triggered. Experimental results have shown that the approach is feasible and effec- tive. It has achieved real-time and reliable alert to avoid collisions of moving ships to bridge piers.
11.1 Introduction
The fact that more and more ships are built while their size becomes bigger and bigger is introducing the high risk of collision between bridge and ship in inland waterways. Incidences of ship-bridge collision mainly cause six types of results, i.e. damage of bridge, people casualty, damage of ship and goods, economical loss, social loss and environmental loss. A large amount of statistical analysis indicates that one of main reasons resulting in ship-bridge collision is the execrable natural environment such as poorly visible conditions, floods, etc. [1, 2].
Mainly, there are two existing strategies to avoid bridge-ship collision at present [3, 4]. One is a passive strategy in which fixed islands or safeguard surroundings are built around bridge piers. The shortages of the passive method are: it could not avoid ship damage from a collision; the costs are normally high; and it becomes less effective with constant increase of ship size. The other is an active strategy that uses radar or video images to monitor moving ships by measuring their course for collision estimation. Compared with the passive method, the active method avoids damage of both bridge and ship and its costs are low. However radar is difficult to detect course changes immediately due to its low short-term accuracy, and the high noise level makes radar sometimes hardly detect any objects from a clutter background. Sensors for visible light do not work well under poorly illu- minated conditions such as fog, mist and night. In contrast, infrared sensors are capable of adapting weather and light changes during a day. Moreover, the FLIR images overcome the problems that radar has, i.e. they have high short-term angle accuracy.
In design, the first consideration of the FLIR surveillance system is its robust- ness for detecting moving ships. The main difficulties are: (1) low thermal contrast between the detected object and its surroundings; (2) relatively low signal to noise ratio (SNR) under the weak thermal contrast; and (3) insufficient geometric, spatial distribution and statistical information for small targets [5].
Motion detection in a surveillance video sequence captured by a fixed camera can be achieved by many existing algorithms, e.g. frame difference, background estima- tion, optical flow method, and statistical learning method [6]. The most common method is the frame difference method for the reason that it has a great detec- tion speed and low computation cost. However the detection accuracy by using this method is strongly affected by background lighting variation between frames. The most complex algorithm is the optical flow method, by which the computa- tion cost is high. The statistical learning method needs training samples which may not be available in most cases, and its computation cost is also relatively high. The background estimation method is extremely sensitive to the changes of the lighting condition in which the background is established.
In the FLIR surveillance video sequence used for moving ship detection, a back- ground normally consists of various information, such as sky, the surface of river, water waves, large floating objects (non-detected objects) in a flooding season, etc. In many cases, ships and background in FLIR images are visually merged together. It is very difficult to detect the targets (moving ships) using normal methods mentioned above.
The new FLIR video surveillance system for bridge-ship collision avoidance is proposed in Section 11.2. Section 11.3 presents the novel infrared image pre- processing algorithm using multi-scale fractal analysis based on the blanket method. The moving ship detection algorithm in the region of surveillance is also developed in Section 11.3. Section 11.4 demonstrates the experimental results with detailed discussion and analysis. Finally, conclusions and future work are presented in Section 11.5.
11.2 The FLIR Video Surveillance System
In the system, a pan-tilt is fixed on bridge pier, and FLIR camera is installed on the pan-tilt. The visual region of FLIR i.e. the region of surveillance, can be adjusted by the pan-tilt, and is configured according to real conditions. The FLIR camera links to a personal computer (PC) through a frame grabber. When images are captured, the image processing program in PC’s memory is used to detect the moving ships. When the moving ships are detected in the region of surveillance, the device for safety alert is started. The ship driver could be alarmed if necessary, and he/she would take maneuvers to avoid ship-bridge collision. The flowchart of the system and sketch map of installation is depicted in Fig. 11.1.
Large amount of experiments carried out in the Yangtse River have proved that the minimum pre-warning distance between bridge pier and ship to avoid collision is 500 m in inland waterway, and the valid distance for moving ship detection is from 800 to 2,000m when the uncooled infrared FPA (focal plane arrays) ther- mal imaging camera is used. Therefore, this type of camera is suitable for the application. The camera resolution is 320 240 pixels. There are three ways designed to trigger the pre-warning signal, i.e. automatically broadcast the pre- recorded voice through very high frequency (VHF), automatically broadcast the pre-recorded voice through loudspeaker, and automatically turn on the assistant lighting system.
11.3 The Detection Algorithm for Moving Ships
In order to detect moving ships in an FLIR image from complicated background along the inland waterway, a novel detection algorithm is developed. It consists of four main stages: extracting the region of interest, i.e. the region of surveillance (ROS); calculating the multi-scale fractal feature; generating the binary image based on the fractal feature; detecting moving ships by a frame difference method. The algorithm is schematically demonstrated in Fig. 11.2.
11.3.1 Extracting the ROS
The ROS is defined based on various conditions in real parts of the inland waterway. Consequently, the ROS appears as a part of region in the original image. The image analysis and processing is focused on this region only. This excludes unwanted regions to reduce computation cost in further processing.
11.3.2 Calculating the Multi-scale Fractal Feature
In practice, ships in FLIR images are treated as man-made objects in contrast to natural background. A fractal model can well describe complex surface structure characteristics for natural objects, but not for man-made objects [7, 8]. To some extent, fractal features of natural background keep relatively stable, but fractal fea- tures of man-made objects behave obviously variety. Therefore, the fluctuation of fractal features distinguishes natural and man-made objects with the variation of scale. The multi-scale fractal feature is proposed to reduce interference of natural background and enhance the intensity of ships.
Many researchers [9, 10] adopted Mandelbrot’s idea and extended it to surface area calculation. For a grey image surface, the fractal dimension can be estimated as in Eq. (11.1).
11.3.3 Segmenting the Fractal Feature Image
C (x, y) provides sufficient information to discriminate natural background and ships. The simplest OSTU segmentation method [11] is used to segment the C (x, y) image. In the resulting binary image, pixel value 255 represents ships and other man-made objects.
11.3.4 Detecting Moving Ships
In the process of moving ship detection, a difference between two binary images generated from segmentation of C (x, y) is used. A group of pixels with non-zero values represent the difference. Based on the fact that the FLIR camera is fixed on a bridge pier, the process is summarized as follows.
11.4 Experimental Results and Discussion
The testing experiments were carried out an FLIR camera was mounted on a bridge pier. A Celeron1.5 Ghz PC was connected with the camera through a frame grabber, the frame size was 320 240, and the frame rate was 30 fps. The parameter settings in the algorithm were, the frame interval as ten frames, the value of threshold (th) as 5, the value of N set as 2, and the value of “max as 4. A group of testing results is demonstrated in Fig. 11.3. The average processing time for each step in the algorithm is shown in Table 11.1.
Observations have indicated that the speed of moving ships is from 20 to 30 km/h. The ROS defines the distance between moving ships and bridge pier as 800 — 2,000 m. Therefore the time during which a ship driver takes action to avoid collision to a bridge pier after altered is between 96 and 360 s. From Table 11.1, it is clearly shown that the FLIR surveillance system takes about one s to complete a process, due to the value N set as 2 or 3. It is satisfactory to the application with a real-time manner.
Comparative experiments were also carried out for system performance analysis in terms of reliability and effectiveness. The frame difference method was imple- mented to be compared with the proposed method. FLIR frames were carefully selected for this comparison. Weather conditions and time of a day were taken into account when 400 frames with 286 moving ships involved were chosen as the testing set. Two parameters were introduced as the criterion for the performance, i.e. false alarm ratio (FAR) and missed alarm ratio (MAR). The comparative results are shown in Table 11.2. Some typical experimental results are demonstrated in Fig. 11.4.
From the results in Table 11.2, it can be seen that the proposed method for bridge- ship collision avoidance is superior to the frame difference method in the criterion of both false alarm ratio and missed alarm ratio. From the results of the testing set and Fig. 11.4, the system is capable of adapting weather and light changes during a day.
It is worth to mention that while the FLIR system is mounted on bridge deck, the performance of surveillance system is impaired by vibration caused by moving vehicles on the bridge. Therefore, the FLIR system is mounted on a bridge pier in practice.
11.5 Conclusion
This paper presented a novel FLIR video surveillance system for bridge-ship col- lision avoidance by using the multi-scale fractal feature, by which moving ships have successfully been separated from the complex background in inland waterway images. The proposed algorithm for moving ship detection has achieved the real- time performance within the ROS in FLIR video sequences. Experimental results have proved that the developed FLIR video surveillance system is efficient in detect- ing moving ships to alert possible bridge-ship collisions. Its wide adaptability and
