Hierarchical Inspection System Using Visual and Mfl Probe Robots

Received Oct 22, 2018 Revised Nov 16, 2018 Accepted Nov 30, 2018 Structural Health Monitoring (SHM) represents a critical appurtenance to modern engineering that amalgamates the skills and techniques from various disciplines of engineering and computational science. Modern civil architectures, involving high-rise buildings, complex structural designs, and innovative shapes, on one hand represent the urban development, but at the same time is a challenge from sustainability perspective. In order to ensure the tenability of such structures, advanced SHM procedures need to be developed. The presented work in this paper is an effort on these lines. The wear and tear in the buildings related to weather, as well as other natural disasters, needs to be monitored regularly and systematically in order to prevent any serious structural damage. In current SHM practices, human experts are deployed at various structurally critical places on these buildings to perform specific measurements and analyze them to decide on the structural health condition. This simple approach is becoming more and more complicated as well as perilous for the human personnel involved, due to the modern architecture that involves greater heights, and complex structures. The proposed system utilizes flying and crawling/roving robots for this purpose. The flying robots, first, scan the surface of the building to any height needed, and then the custom-designed algorithms analyze the images from these scans in order to discern the possible defects/anomalies in the structure. Using these defect pointers, the custom-designed rover robot on top of the structure lowers a robotic probe that scans only those areas for substantiating the anomalies and the degree of defects present. Keyword:


INTRODUCTION
Modern urban skyline is marked by streaks of sky-scrappers with more and more complex designs than before. These marvels of civil and structural engineering are an integral component of the development and progress in a country. However, the size and locations of majority of such structures are exposed to the natural elements and, consequently, have to face various forms of natural disasters, such as storms, strong winds, earthquakes, etc. The strength of such structures can weaken over the years due to wear and tear; hence, a constant health monitoring system is essential for the sustainability and safe operation of these structures. With the rise in natural disasters, as reported by the special commission on disaster and recovery under UNISDR [1], including 6,457 weather related incidents during 1995-2015. These numbers are rising during recent years and has created even more urgent need for better systems for structural health monitoring (SHM) [2]. SHM is not a new area of expertise and many current standards outline a number of procedures

UAV Specification
For the VBI system, the images can be obtained by any means possible as long as the images are obtained with fixed initial reference positions for appropriate registration of successive images. Hence, images can be obtained from static cameras located at pre-defined places [14], or they can be movable [15]. Since the target of this work is related to a perpendicular image scan of the structure in order to develop the best surface scan, the cameras must be as perpendicular to the surface as possible. Hence, a commercially available UAV, Phantom Pro III UAV [20] as shown in Figure 1, has been used for providing the VBI component of the system with its stabilized 4K camera. The high-resolution images from the UAV provide a coarse map of possible changes that have happened during the current and previous scans. The VBI system generates a coordinate set for each defect location, comprising of (longitude {L}, latitude {D}, and height from the base of the structure {H}). This information is passed on to the MFL inspection system once the visual scans are finished.

Hall Effect Based Verification
In the Nondestructive Testing (NDT) industry, two magnetic techniques are very commonly used for specialized testing. The most established method is known as Magnetic Penetrant Inspection (MPI). In this method, the surface to be inspected is first painted with a magnetic ink that spreads evenly on the whole area. Then, hand-held electromagnetic poles (called yoke) are placed around the specific location on the surface and a strong Alternating Magnetic field is generated through them. Due to the vibrational effect of the field, the eddy currents are produced on the surface causing the magnetic particle to align with the field. However, if the surface has any crack or discontinuity, then these particles would concentrate in that defect region due to mutually attractive magnetic dipoles produced in the ink [21]. This concentration appears in a darker color and reveals the crack location. However, performing this test on a huge structure is not practical. The other commonly used technique is called Magnetic Flux Leakage (MFL) based defect scanning, and is one of the most commonly used approaches for inspecting the metallic structures with larger structures such as pipes or tanks [11]. Usually, these devices are used with flowing form factors. This means that the inspection hardware moves in the structure either with the fluid (for pipeline inspection) or with robotic scanners (for tanks). The size and system requirements for such devices render them impractical for most of the structural health monitoring applications being focused in this work. The structural inspection (SI) architecture presented in this paper also includes the in-house design of an alternative solution; a robotic MFL scanning system which is small, flexible, remotely controlled and enables the inspection of difficult or inaccessible structures.
Other than the UAV, the localized and more detailed inspection was done through custom-designed rover robots comprising of two tethered robotic modules as shown in Figure 2. These were named as First Robotic Module (FRM) and Second Robotic Module (SRM) for the sake of simplicity. FRM is the main localization robot which was placed in the beginning of the experiment at the roof of the building. The FRM is responsible for assisting the SRM in performing the actual inspection function through all the necessary computational tasks enabling the SRM to reach the correct height on the vertical surface of the building and perform the inspection. In addition, FRM also acquires the collected data from SRM and communicates between VBI base computer and SRM. The SRM is the designed robotic probe to perform the MFL measurements through custom-designed sensor head and tethered position control through FRM. The SRM has on-board data acquisition hardware that hosts the MFL scanning sensors comprising of Neodymium magnets, Hall-effect sensors, and signal conditioning electronics. The permanent magnets produce very strong magnetic field to generate the necessary magnetic circuit for the sensor operation. At the same time, the magnetic pull also keeps a uniform contact distance and attraction force between SRM and the structure. Actual robotic probe positioning scenario Figure 2 shows the two robots in the MFL inspection system. The FRM, Figure 2(a), has been designed using several off the shelf component modules, and is strong enough to position the SRM in its required inspection location and, at the same time, sustain other environmental effects like winds, altitude, etc. The FRM has precision motion control, GPS coordinates-based localization, obstacle avoidance strategies, and control for correct height selection for the SRM probe robot. The main computing hub for FRM is built around the Arduino platform, Mega 2560 Microcontroller, which is strong enough for FRM-SRM and FRMVBI system communication as well as for localization computations. The SRM is also custom-designed using VEX Robotic kit [22], with possibility of size modification in order to reciprocate the area to be inspected and to be able to move on a variety of structural sections of the target structure. Figure  2(b) shows the heart of the sensing system; the rectangular Neodymium permanent magnets along with the MFL sensor that are placed at the base of the robot. Actual flux sensing is done using the SS495A2 Halleffect sensors. The signals from the sensors were received wirelessly and then recorded in the FRM storage interface to a tablet PC. As shown in Figure 3, the PC provided a graphical user interface (GUI) in LabVIEW environment for easy monitoring and access.

287
The set of coordinates, representing possible defect locations on the surface of the structure that were generated by the VBI system were provided to FRM located on the rooftop of the structure in order to coordinate the motion trajectories for the SRM, based on the corresponding longitude and latitude values. Once reached at the shifted longitude and latitude location due to the building structure (Figure 2(c)), a modified winch motor lowers the robotic probe (SRM) on the surface of the structure to a height that corresponds to M-H, where M is the height of the structure. Once, the probe has reached near the suspected defect location, the sensor robot using the custom-designed MFL sensing head performs the necessary scan in the vicinity of the target location.

Testing Condition and Structure Used
The selection of structure and the testing conditions is one of the critical tasks in the design of this SI system, since it must emulate the main challenges of the real world structures to as greater degree as possible. A test structure was built for performing various initial tests in the laboratory. The structure consists of four homogenous metallic cubes placed sequentially on top of each other, as shown in Figure 4. The reason of selecting such a structure is to ensure that the conclusions obtained from this research address most of the typical civil structure problems without being loaded down with other uncertainties or scaling factors. Several experiments were in this environment in such a manner that the parameters modified by the environment, e.g., wind, camera drifts or zooming issues etc. can be avoided at first and then added in a controlled manner to test the limits of operation of the system.

Sample Defects
The primary aim of this research is to analyze the mechanical defects, such as the structural displacements, cracks, and voids. These kinds of defects are mainly caused in structures by the earthquakes, storms and strong winds. However, these defects are also the major signs of the structural-degradation caused by the passage of time and other environmental conditions. Usually, a subsurface defect is manifested one way or the other on the surface due to the inter-related effects; such as surface cracks due to sub-surface stresses, color changes due to corrosive activity underneath the surface, etc. A good example for such defects can be observed in the structures near sea or in areas where there is significant moisture in air for most of time throughout the year. Such structures will badly suffer with the corrosion, which ultimately results in forming progressing cracks or erosions and can be detected using the proposed method. The sample defects used in the lab experiments, reported in this work, have been induced artificially in the metallic structure so that the changes in the structure can be easily visualized using the proposed inspection system.

PROPOSED APPROACH
The deployment of MFL and VBI subsystems in the above-mentioned hierarchical approach enabled the whole SHM system to function in appropriate period. The coarse-level monitoring was done first with VBI, followed by the detailed and more localized MFL inspection. This provided a good match of larger coverage area and better defect detection. The overall inspection process and the deployment scenario is depicted in Figure 5 [23].

Visual Inspection
The visual inspection entails acquiring two databases of the surface images through the UAV-based vertical scans of the structure. The first database represents images of the structure surface taken under normal, no-defect conditions. This marks the Reference images to be compared with a successive scan with defects induced on the surface of the structure. Images in the first database are named as Images-withOut-Cracks (IoC). Similarly, the images in the second database were named Images-With-Cracks (IwC) since they were acquired after inducing defects, such as imitated cracks, etc. In order to make the images acquisition more similar to the actual image scanning process, the two databases were acquired in slightly different illumination conditions. For both the above acquisitions, the drone was made to fly vertically upwards only from the same fixed location in front of the structure. During the flight, the UAV camera kept taking high quality images at fixed displacements. The flight was kept as vertical as possible such that the GPS coordinates of the UAV at various imaging stops remained the same as that of the take-off coordinates (only Longitude and Latitude are considered). The same vertical strip-scan can be continued until the whole structure is completely scanned.
Once images were acquired, an image stitching (IS) algorithm was applied on each individual set of images in the vertical scans. The IS is comprised of a sequence of hierarchical operations applied on the consecutive/adjacent images of a particular image set. In the algorithmic setting, one of the two images being stitched together was named, reference image (RI), and the adjacent image was named, current image (CI). The resultant stitched image consequently became RI for the next consecutive adjacent image and the same procedures continues until all images in the vertical strip are completed. The process was then repeated horizontally in order to stitch the vertical images with their horizontal neighbors until a complete surface image is obtained. The stitched images were then saved in the database with their corresponding GPS values obtained through the UAV during its flight. Due to its vertical flight only, the variations in longitude and latitude are minimal and average GPS coordinates are used as representative of the completely stitched image. More importantly, the actual structural height (M) is mapped onto the image as well. This is done in order to locate any specific section on the image onto the actual structure.
Although, IS is a fairly standard operation in image processing community, the presented algorithm has tackled a more challenging problem where the images of the same structure are not registered due to the camera movement (being on UAV). The algorithm has integrated two main techniques; Speeded Up Robust Features (SURF) [17], and RANdom SAmple Consensus (RANSAC) [24]. The algorithm starts to stitch the images after locating the Points of Interest (POI) in both RI and CI based on scale and rotation-invariant SURF features. The size of the feature vector (FV) size was set to 64, which was the lower available limit in the SURF function. The common POIs between RI and CI were then determined by computing a distance matrix (DM) for each FV of RI with all possible FVs of CI. The distance less than the defined threshold 289 value (THV) in the resultant matrix was used for determining the FV of CI which was similar to the corresponding vector of RI. In this work, the THV was selected to be a percentage of the computed distance with respect to the perfect match. The term perfect match refers to the least value among all the computed distances for each possible pair of FVs. It is possible that a single DM can have multiple instances where the computed value is less than the THV. Thus, a single FV of RI can have more than one common FVs in CI and/or vice versa. In order to avoid this problem, the forward-backward crosschecking approach was used. In this approach, after matching FV of RI with all FVs of CI, the whole process was repeated in reverse direction by means of matching FV of CI with all FVs of RI. The matches with the least distance were selected as the final common feature points (CFP). For this CFP, an index-matrix (IM) was consequently generated which contains the original indices for each similar FV.

Let
and be the extracted features with respect to the POIs of RI and CI, respectively. Where, 64 represents the FV length, while are the number of POIs for RI and CI respectively. The DM generated for the of RI is shown in (1), where represents the union of all distances of FVs of CI with the of RI. Let L min be the distance of the perfect match then the percentile distance (PD) matrix is generated by dividing each distance value with the L min , as shown in (2). Thereafter, the set of distances with values less than THV were stored in Min Distance as given by (3).
Where y represents the percentage distances with values less than THV. Let P th , Q th and R th FVs of CI; whose corresponding distances with the a th FV of RI satisfy the conditions for Min Distance set, then the FV of CI, which corresponds to the a th FV of RI, can be described as a correspondence set as shown in (4).
Where, similar feature vectors (SFV) is the FVs common between RI and CI. After computing SFV for FRI, the whole process was repeated for FCI such that each FV of RI corresponded to a unique FV of CI in the final CFP. The final IM based on CFP is shown in (5).
It should be noted that indR and indC are the index values of CFP for RI and CI respectively. The matrix IM had a dimension of k × 2 where k is less than or equal to the minimum of i and j. This was due to the fact that not all extracted features were common in both images. Moreover, CFP has a unique pair of FV such that repetition of FV was not permitted for either of the images.
Since the extracted features represented the SURF based POIs, therefore, the POI's of each image with respect to the CFP were determined by using feature points indices in IM. These points were termed as Matched Points (MP) and were shown in (6). The most critical part of image stitching was to align both images and then merge them together. The calculated MPs were then used for orienting CI with respect to RI, similar to a dot product method. However, this could still have a number of outliers, which could produce unwanted overlaps in the image transformation. Therefore, RANdom SAmple Consensus (RANSAC) algorithm [25] was utilized in order to exclude the outliers and compute the homogeneity between the two images [26,18]. RANSAC algorithm estimated the image transformation parameters which best suited the MPs of images. The POIs obtained after performing RANSAC were termed as inliers. These points were pivotal in image stitching because they were used to estimate the transformation parameters, which were required to align both the images. Therefore, it can be concluded that the stitching results depend on the information provided by the inliers. Consequently, greater the number of inliers, the probability to achieve reliable image registration would be higher.
Furthermore, RANSAC is an iterative process and it is generally stated that the probability to get the correct transformation is directly related to the number of trials (NTs) [26]. However, increasing the NTs will also increase the overall computational time of the algorithm. Moreover, it has been observed in this research that the impact of NTs after a certain point becomes negligible. Simulations were performed on a selected pair of images, with THV = 4, five times for each value of NTs and the average values of these five simulations, computed by (7), was taken as the final representative value for the scenario under study. The standard deviation for each simulation was also recorded, using (8), in order to verify the obtained results.

where xs = Parameter value at each simulation; Mean values of all simulations; and K = number of simulations
It was observed that the number of trials also has a limiting effect as shown in Fig. 5, which shows that the obtained number of inliers on each simulation run is showing negligible deviation after 1700 NTs. Furthermore, it has also been observed for the given data that after 1500 NTs, the number of inliers show very small difference of 5 to 7 inliers above and below the average number of inliers as shown in Figure 6. The number of inliers, along with transformation details obtained after the RANSAC operation, was used to determine the orientation of CI with respect to RI. Hence, CI can be easily transformed in such a manner that both RI and CI have same orientation and were also positioned with respect to each other. These newly aligned images for RI and CI were termed as aligned reference image (ARI) and aligned current image (ACI), respectively. In order to fuse the images, a reference-based-blending approach was utilized. In this technique, only those portions of ACI were added to ARI where the corresponding pixel values were zero. This implies that the resultant image would share a major portion from RI. In addition, an external rectangular frame was also added for standardization of the stitched images for further processing. The external frame was defined from top-left corner to bottom-right corner by the corresponding coordinates (0, 0) and (x mf , y mf ). Similarly, the position of ARI in the frame corresponds to (X lr , Y lr ) and (X mr , Y mr ). In the same manner, the transformed image ACI corresponds to (X lc , Y lc ) and (X mc , Y mc ), respectively. In addition, let Z be a set of all pixels in ARI which have non-zero values then the image fusion (IF) was carried out using (9). Figure 7 summarizes the complete image-stitching algorithm as a flowchart.
(9) Figure 7. Image stitching algorithm used in this work, once image fusion is completed, the anomalies were detected using absolute difference of the current and reference fused images At this point, the relative defects can be found by using absolute difference between ARI and ACI images. The probable anomaly locations thus found, were thresholded into a binary image, which provided the locations of these anomalies in the image. Mapping actual height of the structure to the image height results in actual altitude coordinate for the inspection robots in order to deploy the MFL probe at these locations.

The MFL Inspection Process
The GPS module present in the FRM guides the robot to its new desired location and how to reach the vicinity of the anomalies detected by the VBI system. The trajectory control module of the FRM code calculates the resultant trajectory based upon the received GSM coordinates from the VBI system. An offset translation is added to the coordinates because UAV coordinates were outside the building being inspected, while the FRM and SRM robots are on the rooftop of the same building. In order to obtain a correct translation the drone should remain at particular distance from the structure during the scanning process completely. The robot takes its direction based on the comparison result between its current GPS coordinates and the translated received GPS coordinates. Since, there can be several temporary or permanent obstacles present on the roof due to its design. Therefore, an ultrasonic range sensor was also on-board FRM in order to provide obstacle avoidance capabilities to the robot.
As FRM reached the extended target coordinates, a modified winch motor was activated that lowered the SRM with a tethered/wireless data connection to reach the required location of the anomaly. The SRM, a robotic probe with custom-designed MFL sensing system, obtained the MFL measurements of the anomaly location through localized scanning of the anomaly location given by the VBI system. The scan was done on a square area of typically 100cm × 100cm with center on the calculated anomaly location. The readings were sent to the LabVIEW interface present in the Base Computer System for storage and further processing. The procedure of FRM-SRM coordination was repeated as many times as the anomalies identified by the VBI sub system earlier.

Sensor Calibration
In designing the robotic MFL scanning probe, the first challenge was to select the type of magnet to be used along with its appropriate placement so that the performance remain comparable with the commercially available hand-held MFL inspection systems. In common Nondestructive Testing (NDT) practice, an electromagnetic yoke is used with a magnetic penetrant paint for inspection purpose. The magnetic field produced by the yoke is powerful enough to cover a relatively large area; however, the operational power requirement along with the weight of the yoke makes it impractical for scanning high-rise buildings. The proposed robotic scanner can fill this void for such applications.
In this work, a number of neodymium permanent magnets (NPM) with the size of 2.5 × 1.2 × 0.5 cm 3 were used. These magnets are strong enough to be used for inspection. However, the magnetic field of a single NPM was not comparable with the magnetic yoke. Therefore, it was inevitable to determine the appropriate number of permanent magnets (PM), and their distance from the structure surface such that the performance of NPM approaches the standard magnetic yoke field strength. It should be noted, that the distance between the NPMs and structure remains fixed i.e. 2 cm to ensure that the robot will remain in uniform contact with the structure in all experiments. The performance with different number of NPM was compared with the standard electromagnetic yoke using MPI approach. For this purpose, a standard welded iron-steel defect plate was used with a known small crack on the periphery of the weld. The initial testing was performed using the standard electromagnetic yolk by spraying magnetic ink on sample plate such that the particles were accumulated on the crack position as can be seen in Figure 8(a). After analyzing the crack in weld using electromagnet, a pair of NPM were placed around the crack and the same paint was utilized. In addition, the magnetic flux was progressively increased by adding more NPM pairs until the crack became visible to the same degree as in case of the electromagnetic inspection. This led to the use of a stack of minimum of 5 NPM pairs can provide the same impact as that of the electromagnet yoke and can be observed in Figure 8

293
A Hall-effect sensor was then used to detect the same crack with five NPM magnets, in the form of voltage values proportional to the magnetic flux, and the observations are shown graphically in Figure 9(a). The Signal to Noise Ratio of the observed signal was improved through deploying discrete wavelet transforms. The Wavelets used in this work include "symlet" ('sym4'), "daubechies" ('db6'), and "coiflet" ('coif2'). After analyzing all the possibilities of WT on different MFL observations obtained, 'coif2' in both 1st and 2nd decomposition levels produced maximum enhancement and best performance in all cases, as shown in Figure 9(b) and (c), respectively. It can be observed that all the graphs shown in Figure 9 have three peaks. This represents the situation where the robot has passed over the target crack location three times in order to verify that the detection is not due to some other form of uncertainty or artifact, rather it is an actual defect. It should also be noticed that although all peaks represented the same crack, every second peak has lower magnitude value than the previous one and the first peak is the highest peak. This is due to the magnetic hysteresis in the defective metallic surface. When the magnet moves over a region on the plate, it also renders that portion magnetized and in the next successive scan, a magnetic field with opposite polarity tends to resist the magnetic field of the permanent magnets. Thus, the net effect of flux leakage detection by the sensor is reduced.

Sensor Testing
After selecting the appropriate parameters required for the MFL sensor design, extensive testing was done on a standard metallic plate with welding defect, as shown in Figure 10(a). For each scan of the defective area, the COIF2 WT has been applied and a 3-D plot view of the obtained results was reconstructed as shown in Figure 10

SIMULATION RESULTS
The design for VBI and MFL based inspections; both were first individually tested and verified successfully. Thereafter, the coordinated robotic system design based on these two approaches was tested on the sample metallic structure.

Results for UAV
Multiple images were taken using UAV by flying it from the same reference location with respect to the test structure before and after inducing anomalies in specific areas of the structure. These images were subjected to IS algorithm and the final stitched image of IoC and IwC were aligned automatically. Afterwards, the absolute difference was calculated between the aligned images to detect the presence of anomaly.
The selected anomalies in this work were artificial artifacts in the structure namely; an induced displacement between the block of the structure, and a missing screw, as described in the Section 2 (D). The stitched results for IoC and IwC along with the difference between them are shown in Figure 11. It can be easily visualized that both anomalies were successfully identified.

Results for MFL
After detecting the presence of anomaly, the expected height of the defected location was sent to the FRM, which directed the SRM connected to the deployment motor, to move to that location and verify the presence of anomaly using MFL probe. The obtained results are shown in Figure 12. It should be noted that the graph has two peaks in positive direction (i.e. increased magnetic field) while before each positive peak there is a negative peak (i.e. reduction in the magnetic field leakage). The reason of positive peak is that both the defects created an open space, which reduces the material resistance and thus increases the leakage in the magnetic field. Whereas, the negative peak is because of the edge-effect, i.e. the magnetic lines dispersed at the edges, thereby, reducing the MFL values.
It can be observed that the negative peak values of induced gap is more than the missing screw. This is due to the fact that the edges in case of induced gap were not sharp as in the case of missing screw. Since the block was lifted only from the front for inducing the gap, the boundary wall took the form of an inclined edge. Whereas in case of screw, the boundary wall was not changed in its shape and the location of the hole was composed of thin sharp edges.  Figure 12. Structural scan for crack identification using MFL (a) without applying any DWT (b) after filtering the signal using COIF2 and (c) after consecutive filtering using COIF2 along with the demonstration of peaks w.r.t the structure

CONCLUSION
In this paper, a scaled-down version of a promising SHM system has been presented. The system is composed of hierarchical and reconfigurable robotic modules to acquire structural health data to locate and discern anomalies in the structure that might have occurred due to environmental and weather-related factors. From a non-destructive testing perspective, robotic probes could be deployed on the surface as a trivial solution to the SHM problem. However, when the structure is prodigious, then the mere probing solution will not be very practical due to the time it takes to complete the inspection. The presented work has evaded this challenge through a hierarchical inspection approach. In this approach, a visual scan is first performed by UAVs with common reference point of launch over two consecutive occurrences of SHM scans. UAVs can be configured for and specific height and imaging quality, however, the presented work has shown initial test results based on commercially available UAV with 4K imaging system on-board. The custom-designed intelligent stitching algorithm aligns and registers these consecutive-run images and, after comparison of the images, identifies the locations of possible anomalies. This significantly reduces the search space for the NDT probe robots to locally-scan these locations for detailed defect elucidations. The lab-scale initial design has shown very promising results. Tests were also performed on two-story building as well for scalability confirmation; and the technique has been found to produce equally reliable defect detection.
Several computational techniques have been amalgamated into the underlying system presented in the paper. Robotic design and control was done entirely based on the problem-specific scenarios. This way, even the commercially available modules, were integrated into the robotic rover and robotic probes to perform the imperative inspection functions. On the soft side of the design, well-known image registration algorithms; SURF and RANSAC, were combined with modified image transformations in an amicable manner that resulted in extremely aligned images with precise defect localizations based on spatial gradients of the registered images. Moreover, in this paper, different WTs have been applied on the MFL waveforms in order to improve the signal to noise ratio and, more importantly, distinguish the undistorted signals. It has been concluded experimentally that by applying COIF2 on two consecutive stages will provide the most refined output waveform, suitable for defect detection and, to some extent, its depth grading.