HD4AR

A construction project requires a large amount of cyber-information, such as 3D models.  Unfortunately, this information is typically difficult for construction field personnel to access and use on-site, due to the highly mobile nature of the job, as well as a hazardous work environment.  Field personnel rely on carrying around large stacks of construction drawings, diagrams, and specifications, or traveling to a trailer to look up information electronically. We present Hybrid 4-Dimensional Augmented Reality (HD4AR), a mobile augmented reality system for construction projects that provides high-precision visualization of semantically-rich 3D cyber-information over real-world imagery. HD4AR allows construction field personnel to use mobile devices, such as a smart phone or tablet, to take pictures that include a specific construction element, see BIM elements visually overlaid on top of the real-world imagery, touch or click on a BIM element in the image, and be presented with a detailed list of cyber-information, such as plan information (e.g., budget, specifications, architectural/structural details) or actual information (e.g., cost, safety provisions, physical progress) related to the physical element.

HD4AR: Hybrid 4-Dimensional Augmented Reality

Problem:

How can you create physical boundaries for where information can be accessed on mobile devices?

Context:

In 2010, the US spent 816 billion dollars on construction projects. It is estimated that a 0.1% improvement in efficiency of project delivery could save roughly $200 million. Even though there are significant potential cost savings, the construction industry still faces major challenges in tracking and monitoring operations to promptly identify, process, and communicate discrepancies between actual and expected performances. The National Academy of Engineering has identified "enhanced visualizations, through advances in virtual and augmented reality" as a key challenge for 21st century engineering.

One of the main challenges during a construction project stems from the inability to effectively access, interact with, and visualize various types of information (such as construction drawings, specifications, and expected progress information) in the context of the actual physical elements the information is related to. The quality and timing of transferring or exchanging hundreds to thousands of pieces of information can either delay or facilitate the execution of a project during the construction phase.

To successfully control execution of a project, the actual (physical) status of the infrastructure under construction needs to be constantly monitored and compared with the project’s cyber-information model, consisting of elements such as structural geometries, their spatial and material properties, which can be augmented with cost and schedule information. Early detection of deviations between the physical construction site and expectations captured in the cyber-information can facilitate corrective decision-making and reduce cost.

Despite the importance of cyber-physical information association on a construction site, current monitoring practices include manual and time-consuming data collection (e.g., walking around a construction site and writing down information on paper), non-systematic analysis (e.g., rough comparisons of the notes to the 2D construction plans), and visually/spatially complex reporting (e.g., estimating how real-world 3D physical structures correspond to 2D building plans or virtual 3D models).

Open Problem:

A key issue with these manual processes is that practitioners cannot access and interact with cyber-information through the physical structures that they are building. For example, there is no way for a field engineer to see a 3D plan for a wall overlaid directly on top of the physical element. Instead, field engineers must either carry bulky stacks of drawing and documents, or make trips back and forth to construction trailers and offices to look up plans for building elements and compare them to what was seen on the construction site. Because there is no easy way to visualize and query the cyber-information through the actual physical construction elements, it is difficult for field personnel to quickly detect discrepancies between the construction elements and the building plans. Without quick identification of discrepancies, managers cannot easily adjust the construction plan to minimize the impact of problems.

Currently, mobile devices, such as smartphones, are commonly available on construction sites, making the construction zone a prime target for mobile augmented reality. Mobile augmented reality (AR) is an approach whereby a mobile system enhances a user’s perception of their environment by mixing the physical view of the world, captured with a mobile device’s camera, with computer generated visual information shown on the camera’s preview display. The rise in popularity and availability of smartphones has led them to the forefront of mobile augmented reality. This is because smartphones unite many of the components of an AR system into a single unit. They have a camera, color display, and a variety of sensors, making it very easy to eliminate the need for custom hardware.

However, while smartphones provide the platform and tools for AR, existing mobile AR approaches are not well suited for a construction environments due to their lack of accuracy in spatially locating the user and rendering cyber- information over the correct physical elements. On a construction site, discrepancies between physical elements and the cyber-information must be detected to within a few centimetres or less. AR approaches, based on GPS and compass sensors, can have multiple meters of inaccuracy, making them unreliable for construction scenarios.

Solution Approach

 HD4AR is based on a computer vision approach that uses image feature detection, matching, and Structure from Motion (SfM) algorithms to extract 3D geometry from a set of overlapping photographs. Initially, HD4AR is bootstrapped by extracting 3D geometry from daily construction photographs using an offline process. Once the system is bootstrapped, field engineers can take new photographs with a mobile device, wirelessly upload them to an HD4AR server, have them localized in the extracted 3D geometry, and then receive an augmented photograph with BIM elements precisely overlaid on the real-world imagery. Our empirical results show that this end-to-end process can be completed in roughly 60 seconds or less.