Introduction

In cultural heritage, proper storage, organization and classification of digital heritage data via mathematical logics is a key issue. More automated annotation tools are demanded to adequately support different experts in heritage conservation, preservation and documentation processes in recording, archiving and classifying existing digital information, even starting from raw, unorganized surveying data.

In this light, the contribution presents several significant results of a Ph.D. thesis work, defended in June 2022, that aims at defining a methodological approach for the proper transfer, retrieval and exchange of semantic annotations on 2D/3D digital models of cultural heritage, based on the combination of Artificial Intelligence-based algorithms for semi-automated annotations, Heritage-Building Information Modeling (H-BIM) systems and collaborative reality-based annotation platforms.

In the three respective phases of the proposed approach, i.e., segmentation via Artificial Intelligence (i), 2D/3D information propagation (ii), H-BIM reconstruction and semantic structuring (iii), novel interactions between digital heritage and mathematics are differently investigated.

The Research

In the digital heritage domain, the mechanism of semantic annotation, understood as the appropriate association of semantic (knowledge-related) information to purely metric digital data (Abergel et al. 2021), is essential to ensure correct interpretation and sharing of heritage information over different types of digital representations (e.g., meshes, point clouds, images, CAD drawings) (Ponchio et al. 2020). However, the semantic annotation and enrichment of data that are derived, e.g., from laser scanner or photogrammetric surveys, are at present largely manual processes, therefore subjective and prone to errors (Brumana et al. 2020). Moreover, a plurality of digital representations might be used for the annotation process, ranging from 2 to 3D (as H-BIM) to hybrid reality-based annotation platforms as Aioli (aioli.cloud), and this multiplicity of supports being annotated may lead to possible data dispersion or data alteration issues (Croce et al. 2020).

In this context, the appearance of new tools for automatic annotation, based on statistical learning and Artificial Intelligence, opens the way to new horizons of interaction between cultural assets, digital technology and mathematical methods for data interpretation and organization (Croce et al. 2021; Morbidoni et al. 2020). Semantic, mathematic and morphological issues can indeed be put together within a knowledge model, so as to access new levels of analysis, survey processing and information management (Bevilacqua et al. 2018).

Considering Heritage-Building Information Modeling (H-BIM) systems and reality-based collaborative annotation platforms such as Aïoli (Manuel et al. 2016) as a starting point toward archiving and sharing semantic information, the objective of this work is to propose a methodological approach allowing the transfer, retrieval and exchange of semantic annotations over 2D/3D digital heritage models. An adequate method for sharing, updating, and transferring information, to be valid regardless of the type of digital model chosen for the representation, would meet the need to archive, share and update data related to a given heritage artifact.

Aiming at this objective and based on input 3D survey and analysis data, the proposed methodological approach is developed according to the three respective phases of:

  1. i.

    Semi-automatic semantic segmentation (classification) of surveying data through Artificial Intelligence-derived algorithms.

  2. ii.

    Annotation transfer via reconstruction of the geometrical-projective relationships linking 2D and 3D representations.

  3. iii.

    H-BIM reconstruction, semantic structuring via shape grammar and insertion of localized information.

In detail, the semi-automatic semantic enrichment of digital data is investigated first by application of Artificial Intelligence algorithms, and of Machine Learning in particular, to enable more automated interpretation and classification of raw surveying data (either point clouds, images, meshes and so on), based on to the recognition of architectural components, the detection of degradation patterns, the material mapping, and so forth (Figs. 1 and 2).

Fig. 1
figure 1

Semantic segmentation via AI on input surveying data

Full size image
Fig. 2
figure 2

Classification of 3D digital heritage toward more automated Scan-to-BIM reconstruction techniques

Full size image

In this first phase, semi-automated systems are exploited to better support 2D and 3D interpretation of raw data and characterization of the surveyed architectural scene in terms of recognition of forms, materials, state of conservation of the heritage object. AI-based techniques are considered here for the construction of appropriate tools supporting the processing, management and semantic enrichment of digital heritage objects. Starting from the extraction of appropriate geometry-based and texture-based features, and by providing a relevant set of training data, an artificial intelligence is trained to classify, within digital models: the architectural elements that make up a scene (distinction of main architectural components of a building as roof, molding, columns, vaults etc.); the thematic mapping of degraded surfaces; damage maps and specification of intervention types; virtual reconstruction of lost elements and organization of archival documentation; definition of materials and their assembly, construction techniques, etc. The semantic classification phase via ML thus allows the acquisition of 2D or 3D data that are semantically annotated, hence more easily interpretable. The methodology is studied both with reference to 2D data (photos, ortho-photos, drawings, archival images) than to 3D models (meshes, point clouds) of the architectural heritage and leverages supervised ML algorithms. In detail, the Random Forest algorithm (Breiman 2001) is considered as main statistical Machine Learning model, while geometric and radiometric features are extracted from input surveying data, starting from mathematical formulas, to support proper classification of raw surveying data.

At a second stage, the information obtained is transferred and propagated to multiple representation systems, from 2 to 3D and vice versa, also through the use of the Aïoli collaborative annotation platform (Manuel et al. 2016); geometrical interactions are exploited to numerically reconstruct the projective relationships that link together different types of digital representations, as images and 3D models. In detail, new modalities of semantic annotation transfer and propagation are investigated, starting from the geometrical, mathematical reconstruction of the relationships linking 2D images to 3D models and vice versa. In this context, for instance, the transfer of semantic data between meshes and images or between point clouds and images, or again between 3D models at different resolutions, is explored. The aim is to provide a possible future user with a range of feasible methods to be chosen as most suitable for data transfer, according to the case-specific needs and requirements for semantic data enrichment (shifting, e.g., from point clouds and meshes to images, UV maps, ortho-photos and vice versa). Multi-layer and multi-resolution workflows are later offered to enable the transfer of semantic data at different levels of detail.

The hybrid annotation system proposed by the Aïoli platform, relying on 2D-3D semantic data propagation, is even combined with more automated annotation methods, making the most of Artificial Intelligence and statistical learning. As such, an implementation of the Aïoli platform with semi-automatic (AI-driven) annotation methods is presented, so to improve the possibilities of propagation of semantic data to date available in Aïoli.

At a final stage, the classification of a heritage model by AI, when guided by the distinction of recurring architectural components (e.g., column, vault, molding etc.), is exploited in view of the construction of H-BIM models starting from annotated survey data, to automate the so-called Scan-to-BIM process. In other words, classified point clouds are leveraged to reconstructed, based on annotated 3D data, the conceptual geometries of each class of architectural elements detected. By referencing architectural treatises and exploiting shape grammar and generative modeling environments, a set of graphical rules and conventions is defined for the reconstruction of template geometries and for the subsequent propagation of these latter to all elements belonging to a same typological class (Fig. 3). The operation, if performed on the whole model, allows realizing a complete information system of the H-BIM type, that preserves the connection with the original surveying data, built up based on the shape grammar of architectural components, and in which knowledge-related data can be stored.

Fig. 3
figure 3

2D/3D information propagation and H-BIM reconstruction

Full size image

The digital model so obtained results in a semantically structured representation, where the insertion of localized annotations can further be studied for restoration, conservation and dissemination purpose. The experimental context of this thesis involves significant Italian and French case studies, as this Ph.D. work is developed in the framework of an international agreement for joint research doctoral thesis (co-tutelle), involving the Universities of Pisa and Florence (Italy), the research unit ‘Models and simulations for architecture and heritage’ of the French CNRS and the Ecole Nationale Supérieure d’Arts et Métiers ParisTech. The different phases of the proposed approach are tested and assessed with reference to representative case studies of both the Italian and French architectural heritage (Fig. 4): among them, the Notre-Dame Cathedral in Paris and the Pisa Charterhouse. Each time, the results are presented and evaluated by considering the case-specific representation and restitution needs and requirements.

Fig. 4
figure 4

Several significant case studies of Italian and French architectural heritage assets

Full size image

Conclusions

The thesis provides several tools for the realization and sharing of semantically rich digital models, of monuments, historical buildings, sculptures and works of art in general. The proposed tools can be applied in archaeology, architecture, civil engineering and art history, and be adapted each time, depending on case-specific needs of insertion, transfer and semi-automatic archival of semantic annotations. They enable:

  • Classification, interpretation and management of raw surveying data and digital heritage information.

  • Comprehension of the object, by restitution of mathematical, geometric attributes;

  • Multi-modal connection of 2D-3D digital sources of a same object;

  • Collaborative management of information over digital platforms (Aioli or BIM-based);

  • Construction of template 3D models, to be leveraged as H-BIM systems for further semantic enrichment.

The approach will be made available to engineers, architects, historians, and other experts who continuously deal with the issues of collection, fusion and updating of heterogeneous information over digital representations of heritage artifacts.