Introduction

Additive manufacturing (AM) is a transformative technology that is already providing game-changing, new capabilities across a wide range of material systems and applications. For example, polymer AM allows “mass customization” by producing components or parts directly from 3D files. For metals, AM enables production of three-dimensional parts with geometries that can be too costly, difficult or, in some cases, impossible to produce using traditional manufacturing processes. In many cases, however, difficulties persist regarding throughput, reliability, and the properties of the printed parts. In thermoplastic polymers, the strength of the material is limited by incomplete thermal fusion and voids. For metal AM methods that use local melting and re-solidification, the extreme, location-dependent, thermal conditions produced during the AM process often create parts with unexpected phases, steep compositional gradients, and high residual stresses. Post-process heat treatments based on nominal compositions and equilibrium phase diagrams are ineffective for many alloys and can produce parts with severely degraded performance and unexpectedly short service life. The strong dependence of the local properties of AM parts on specific local conditions during the build process makes accurate computer modeling a critical aspect of the AM part design process. The combination of a rapidly changing technology, a lack of material property data, and the need for accurate computer simulations has exacerbated the need for traceable standards and benchmark measurements.

In response to this need, a framework for developing international benchmark measurements for the AM community was proposed by the National Institute of Standards and Technology (NIST) at an AM workshop at the National Academies in Washington D.C on October 7, 2015 [1]. The response was immediate; within weeks, researchers from dozens of organizations around the world volunteered their time and input to make the Additive Manufacturing Benchmarks (AM-Bench) test series a reality. Today, the AM-Bench Steering and Scientific Committees include over 80 scientists and engineers from more than 60 organizations, including NIST, five Department of Energy (DOE) laboratories, six National Aeronautics and Space Administration (NASA) centers, three Department of Defense (DOD) laboratories, and numerous companies, universities, and other research centers in North America, Europe, and Asia. The first round of benchmark measurements has been completed, and the first AM-Bench conference, AM-Bench 2018, was held at NIST, Gaithersburg, MD, during the week of June 18, 2018. Detailed descriptions of the 2018 benchmark measurements, the 2018 measurement results, the AM-Bench organization, and the 2018 award winners may be found on the AM-Bench web page, www.NIST.gov/AMBench. Formal papers about the AM-Bench measurement results along with invited papers from AM-Bench 2018 participants can be found in this special issue of the TMS publication, Integrating Materials and Manufacturing Innovation (IMMI). We would like to thank the many excellent scientists who volunteered their time, effort, and resources to complete these 2018 AM-Bench benchmark measurements. In all, more than 40 scientists participated in the measurements, from eight institutions including NIST, the US Naval Research Laboratory, the US Army Research Laboratory, Argonne National Laboratory, Los Alamos National Laboratory, Cornell High Energy Synchrotron Source, University of California Davis, and Hill Engineering.

In the following sections, we will (1) briefly describe the AM-Bench scope, (2) provide an overview of the 2018 measurements and modeling challenges, (3) give a general description of the blind modeling submissions we received, (4) make general comparisons between the measurements and model submissions, (5) summarize input we received during the AM-Bench 2018 discussion session, and (6) briefly describe the AM-Bench data curation approach. We will then give our conclusions and our vision for future benchmark measurements for AM.

AM-Bench Scope and Benchmark Measurement Selection Criteria

AM-Bench was established to develop benchmark measurements for all AM processes and materials. However, the requirement for highly controlled and quantitative measurements severely limits the number of benchmark measurements that can be conducted in any given test cycle. Thus, one of the key challenges faced by AM-Bench is to select benchmark measurements that are both feasible and have the highest impact on the AM community.

The scope of AM processes and materials currently in use and being developed is extremely broad, complicating the selection process. For metals, AM methods include powder bed fusion (laser and electron beam), direct energy deposition (laser, electron beam; powder, wire fed), binder jet (infiltration, consolidation), sheet lamination (ultrasonics), and many others. For polymers, the most common techniques include material extrusion, powder bed fusion (laser), material jetting, binder jetting, vat photopolymerization, and sheet lamination. AM is also used for ceramics, composite materials, and even for large-scale building construction using concrete.

The selection process for the first round of AM-Bench measurements, AMB2018, had two main criteria. First, benchmark measurements can only be conducted if sufficiently precise and accurate instrumentation and techniques exist to adequately characterize the given process. For example, direct energy deposition (DED) using electron beams was considered, but input from experts in this area indicated that benchmark measurements for DED would not be feasible in the necessary time frame. Similarly, vat photopolymerization is a widely used method that creates polymer parts with good dimensional tolerances. However, in situ process monitoring for this method was not well developed. On the other hand, highly quantitative systems for laser powder bed fusion (LPBF) of metals were either already available [2, 3] or were close to coming online [4]. The second main criterion was that the benchmark measurements must target a material and process of high impact to the international AM community. Since polymers and metals are currently the most heavily used and researched material systems for AM, these were selected for AMB2018. We are grateful for the valuable input and suggestions we received from the AM-Bench Scientific Committee on all these criteria.

For metals, we identified two heavily used AM material systems that were suitable for benchmark measurements, nickel-based superalloys and martensitic stainless steels. To minimize complexity, we selected the simplest nickel-based superalloy in common use for AM, IN625. This nominally single-phase material had the added advantage that many of the AM-Bench scientists participating in these studies had extensive AM experience with it, including 3D builds, in situ monitoring, laser track studies on IN625 bare plate, and microstructure characterization and modeling [5,6,7,8,9,10]. Our initial choice for a martensitic stainless steel was 17-4, but its high sensitivity [11] to dissolved nitrogen (nitrogen is an austenitic stabilizer) from the atomization process made this material inappropriate for benchmark measurements. We instead selected the closely related 15-5, which has a microstructure that does not exhibit this high sensitivity to nitrogen. Austenitic stainless steels are also of interest, but again, most of the participating scientists had more extensive AM experience with martensitic steels, so austenitic steels will be considered for future benchmark measurements.

Many modeling efforts for metal AM include high fidelity models of the melt pool that provide critical input into larger time and length scale models required for full 3D AM builds. AM-Bench 2018 therefore included in situ and ex situ measurements of both complete 3D builds (AMB2018-01) and individual laser melt tracks on bare metal plates (AMB2018-02).

For polymers, the focus was on thermoplastic AM technologies and we identified the two most widely used approaches for AM-Bench: materials extrusion (MatEx) and LPBF, also sometimes referred to as selective laser sintering (SLS). In both cases, the underlying technology is based on the melting of thermoplastics polymer followed by a welding/coalescence step and then solidification. A deficiency in the processes identified by many practitioners is the presence of weak interfaces at these junction points, due to poor molecular interdiffusion and/or voids.

The AM-Bench measurements for polymers were designed to probe such mechanical properties of the printed parts with concomitant characterization of underlying materials properties. AMB2018-03 was based upon MatEx of polycarbonate (PC), in which a sheet was fabricated and strips were extracted for tests of mechanical properties and voids. AMB2018-04 was based around LPBF of polyamide 12 (nylon 12); a series of dog bone-shaped specimens were fabricated along various axes of the build chamber for similar tests. In polymers AM, the state of the art of modeling is somewhat segmented and so we did not expect that many modelers would be able to bridge the full span from molecular characteristics of materials, through processing to final part properties. Thus, the 2018 benchmark measurements for polymer AM have largely preceded the ability of the modeling community to make full use of them. Also, future iterations of AM-Bench measurements for polymers AM will likely give greater attention to lower-level processes.

The final set of AMB2018 materials and processes for each of the 2018 benchmark test categories is listed in Table 1.

Table 1 Materials and processes used for the 2018 benchmark measurements
Full size table

2018 AM-Bench Measurements and Modeling Challenge Problems

The AMB2018 benchmarks covered a wide range of materials, build methods, and measurements. AMB2018-01 included LPBF builds of 3D metal alloy (IN625 and 15-5) test objects with multiple types of in situ and ex situ measurements. The primary focus was on residual stress, distortion, and microstructure. AMB2018-02 included single laser scan tracks on bare IN625 plates (no powder). The primary focus was on in situ melt-pool length and temperature characteristics, ex situ melt-pool cross section and microstructure, and the 3D topography of the solidified tracks. AMB2018-03 included MatEx 3D builds of PC test objects. The main focus was on part thickness, mass, tensile properties, void distribution, and cross section. AMB2018-04 included LPBF 3D builds of nylon 12 dog-bone specimens, built at different orientations with respect to the build axis. The primary focus was on tensile properties, void distribution, and local anisotropy of semicrystalline phase and morphology.

AMB2018-01

AMB2018-01 consisted of LPBF 3D metal alloy builds of a bridge structure geometry that has 12 legs of varying size, as shown in Fig. 1. A primary goal of AMB2018-01 was to provide a high fidelity set of coordinated measurements and process data, covering the full range from feedstock material to finished part. To our knowledge, no such detailed and multifaceted AM study had ever been attempted. AMB2018-01 included comprehensive characterization of the feedstock materials (https://www.nist.gov/ambench/amb2018-01-description), detailed description of the laser scan pattern with sub-ms-level timing [12], in situ thermocouple measurements within the build chamber [12], in situ measurements of cooling rate and melt-pool length in every layer within the field of view (as illustrated in Fig. 1) [12], ex situ characterization of the resulting residual strains and stresses [13], part distortion measurements after cutting part of the test piece off the build plate [13], multiple synchrotron X-ray and laboratory-based microstructure studies as a function of position and orientation within the build [14, 15], and measurements of the phase evolution during residual stress annealing [15]. Correlations between the in situ thermal measurements and the local residual stresses and microstructures should provide a rich proving ground for modelers and greatly enhance insight into how the microstructures, stresses, and properties evolve during LPBF builds.

Fig. 1
figure 1

The AMB2018-01 bridge structure geometry

Full size image

The in situ measurements were taken on two LPBF machines, and test artifacts were fabricated from two commercial alloys: nickel-based superalloy IN625 and 15-5PH stainless steel. The LPBF machines were the NIST-built additive manufacturing metrology testbed (AMMT) and an EOS M270Footnote 1 with modifications for in situ measurements. The EOS M270 will be referred to using the designation CBM, for commercial build machine.

The modeling challenges fell into three areas: the residual strain field within the part prior to the EDM cut, the distortion of the part after the EDM cut, and the as-built microstructure (prior to any post-processing heat treatment) in select regions of the part. One challenge also involved the evolution of the microstructure during stress relief. The AMB2018-01 modeling challenges were:

Part Deflection:

CHAL-AMB2018-01-PD [13]

Residual Elastic Strain:

CHAL-AMB2018-01-RS [13]

Microstructure:

CHAL-AMB2018-01-MS [14]

Phase Fractions:

CHAL-AMB2018-01-PF [15]

Phase Evolution:

CHAL-AMB2018-01-PFRS [15]

Complete descriptions and measurement results for AMB2018-01 can be found on the AM-Bench web page and in the specific benchmark measurement papers in this special issue of IMMI [12,13,14,15].

AMB2018-02

AMB2018-02 consisted of individual laser scan tracks made on bare nickel-based superalloy IN625 metal surfaces. The main objectives were to characterize the 3D geometry of the melt pool after the process reached steady state, the surface temperature as a function of time immediately after solidification, the 3D surface topography of the melt trace, and the microstructure. Three different power and speed combinations were used for the tracks with multiple replications. In situ measurements of the melt-pool length and the cooling rate of the solidifying material were taken. Figure 2 presents an illustration of a substrate, the field of view (FOV) of the in situ measurement system, and a series of ten scan tracks. A significant amount of time (at least 5 min) was allotted between each scan to ensure that the substrate temperature at the beginning of each scan is equal to the ambient chamber temperature.

Fig. 2
figure 2

Schematic of IN625 substrate and nominal positions of scan tracks

Full size image

Bare plate scans were performed on two laser powder bed fusion systems, the AMMT and the CBM. While the scans performed on each system are nominally similar, each system has unique measurement capabilities. A comparison of both AMMT and CBM thermal imaging systems, as well as a thorough description of the measurement methodology for melt-pool length and cooling rates, is provided in [16].

The AMB2018-02 modeling challenges fell into four areas: the geometry of the melt pool, the cooling rate at the top mid-plane of the melt pool, the 3D shape of the solidified melt track, and the microstructure. The specific challenges were:

Melt-Pool Geometry:

CHAL-AMB2018-02-MP [16]

Cooling Rate:

CHAL-AMB2018-02-CR [16]

Surface Topography:

CHAL-AMB2018-02-TP [17]

Grain Structure:

CHAL-AMB2018-02-GS [14]

Solidification Microstructure:

CHAL-AMB2018-02-DM [14]

3D Microstructure:

CHAL-AMB2018-02-3D

Complete descriptions and measurement results for AMB2018-02 can be found on the AM-Bench web page and in the specific benchmark measurement papers in this special issue of IMMI [14, 16, 17]. The 3D microstructure measurements for AMB2018-02-3D were a stretch goal using new, world-leading instrumentation. Due to multiple equipment and infrastructure issues, these measurements were not completed in time for incorporation into the current online database or this special issue. At the time this paper was submitted, these measurements have been completed and analysis is ongoing for eventual incorporation into the AM-Bench 2018 benchmark measurement data set [18].

AMB2018-03

The AMB2018-03 test object consisted of a rectangular prism of PC. The rectangular prism was cut from a sheet that was processed by MatEx technology (sometimes referred to as fused filament fabrication or by fused deposition modeling). The main objectives of the AMB2018-03 measurements were to characterize the primary macroscopic aspects of the print (thickness, mass), the microscopic shape (voids), and the tensile properties.

A Stratasys Fortus 400mc materials extrusion system was used to fabricate PC sheets. The build envelope for the Fortus 400mc is 406 mm × 406 mm × 356 mm (16″ × 16″ × 14″). A T-10 tip, with nominal inner diameter of 0.13 mm (0.005″), was used for the material extrusion. The PC samples were deposited onto SR-100 sheets of nominal thickness = 1.27 mm (0.05″), a proprietary support material. A sheet of the support material was deposited via MatEx prior to the PC deposition; after PC deposition, the support material layer was peeled from the PC at room temperature.

PC sheets of dimensions 177.8 mm × 177.8 mm × 1.91 mm (7″ × 7″ × 0.075″) were processed via materials extrusion. Single sheets were fabricated consecutively on the build plate, as shown in Fig. 3. Sheets were processed using two separate raster orientations: 0° (from which 0° and 90° tensile specimens were cut) and 45° (from which 45° specimens were cut).

Fig. 3
figure 3

Layout of part fabrication on fused deposition modeling platen. All dimensions are in inches (1 inch = 2.54 cm)

Full size image

The AMB2018-03 modeling challenges were:

Part Thickness:

CHAL-AMB2018-03-Th

Part Mass:

CHAL-AMB2018-03-Ma

Part Tensile Properties:

CHAL-AMB2018-03-TP

Part Void Distribution:

CHAL-AMB2018-03-VD

Part Cross Section:

CHAL-AMB2018-03-CS

Complete descriptions and measurement results for AMB2018-03 can be found on the AM-Bench web page.

AMB2018-04

The AMB2018-04 test object consisted of a dog bone-shaped part produced by LPBF of a polyamide powder. The feedstock material was PA12 powder (sometimes referred to as nylon 12 powder). The main objectives of the AMB2018-04 measurements were to characterize the primary macroscopic aspects of the print (thickness, mass), the microscopic shape (voids), the tensile properties, and the crystallinity of the printed part.

Parts were built on a ProX500 SLS machine manufactured by 3D Systems Corporation. Heat was provided by a six-zone PID-controlled array of eight infrared heating lamps above the build chamber, as well as the heated piston and cylinder supporting the build plate. PA12 powder was applied in 0.1 mm thick layers via a compressive counter-rotating roller. The entire build was conducted in a nitrogen gas environment held at 167–169 °C. The system used a 100 W CO2 laser operating at 62 W and rastered at a speed of 12 m/s to form the part slices with a scan spacing of 0.2 mm. The dog-bone parts were built using a range of angles with respect to the build axis, as shown in Fig. 4.

Fig. 4
figure 4

Build layout used for polymer LPBF (background squares are 1 cm)

Full size image

The AMB2018-04 modeling challenges were:

Part Thickness:

CHAL-AMB2018-04-TH

Part Mass:

CHAL-AMB2018-04-MA

Part Tensile Properties:

CHAL-AMB2018-04-TP

Part Void Distribution:

CHAL-AMB2018-04-VD

Part Cross Section:

CHAL-AMB2018-04-CS

Part Crystallinity and Melting:

CHAL-AMB2018-04-CM

Complete descriptions and measurement results for AMB2018-04 can be found on the AM-Bench web page and in the specific benchmark measurement paper in this special issue of IMMI [19].

2018 AM-Bench Modeling Challenge Problem Submissions

The long-term impact of AM-Bench resides in the measurement data which are permanently archived and freely available online. Since AM combines material processing and shaping into a single step, the material structure and properties can vary considerably depending upon the local build conditions. Computer simulation has therefore become the essential tool required to bridge this technological gap. Integrating and validating the wide variety of computer simulations and models over the needed time and length scales for AM require both specific metrics and validated data to gauge their accuracy, and AM-Bench is designed to provide both through a continuing series of controlled benchmark tests, in conjunction with a conference series, to allow modelers to test their simulations against rigorous, highly controlled additive manufacturing benchmark test data. The intent is to create a single set of rigorous benchmark tests that developers can compare against, and to produce permanently archived data that will benefit the entire AM community.

AM-Bench is also providing a substantial shorter-term impact by creating a snapshot of existing computer simulation capabilities for AM, by soliciting modeling results before the measurement results are released (blind submissions). For AMB2018, detailed descriptions of the modeling challenges were released on February 20, 2018, and blind modeling submissions were due by May 18, 2018, three months later. Posting of the measurement results on the AM-Bench web page commenced the next day, May 19, 2018. To encourage individuals and organizations to participate in these blind submissions, AM-Bench 2018 Modeling Awards were given for those submissions that the AM-Bench 2018 Organizing Committee judged deserving, using success criteria discussed in the next section.

In all, 46 modeling challenge submissions were received from 19 groups in eight countries: nine groups in North America, six in Europe, and four in Asia. Some groups included participants from multiple countries. In those cases, the country of the submitting author was used. Figure 5 shows the number of submissions received for each of the AMB2018 modeling challenges. Here, the first column is a graphic indicating the general topic of the challenge, the second and third columns specify each benchmark challenge, and the fourth column shows the number of modeling submissions for each challenge. The colored text designates if at least one submission met the success criteria as determined by the Organizing Committee and the respective measurement teams. Green indicates at least one submission met these criteria, red indicates no submission was successful, blue means that the corresponding measurements were delayed by equipment and facility issues, so an evaluation was not possible. Black indicates that no submissions were received.

Fig. 5
figure 5

AM-Bench 2018 challenge submissions. The first column is a graphic describing each set of benchmark measurements. Columns 2 and 3 give descriptive names and designations, respectively, for all the challenges. Column 4 gives the number of submissions for each challenge or group of challenges. The blue ovals show the groupings for AMB2018-01 and AMB2018-02. The color code for column 2 designates whether at least one submission met the criteria for being successful. Green = successful, Red = unsuccessful, Blue = evaluation pending, Black = no submissions

Full size image

The most obvious feature in these data is the large difference in modeling challenge submissions between metal and polymer AM: 45 submissions for metals and just one for polymers. A significant difference was expected since the scientific community is currently investing much greater efforts in understanding the mechanistic and material basis for metals AM than for polymers. Nevertheless, the magnitude of the difference was larger than expected. It is also worth noting that the low number of polymer AM challenge submissions does not mean that these benchmarks are not needed. Instead, as theoretical and computer simulation work expands in this area, these benchmarks will provide the rigorous measurement data needed to guide and validate such models. It is also possible that the ready availability of such data will stimulate additional work on understanding and simulating polymer AM processes.

Another interesting feature in these data is that AMB2018-02 received twice as many submissions as did AMB2018-01. As mentioned above, AMB2018-02 provided benchmark data for the basic, underlying physics of melt-pool formation and the concomitant development of the solidification microstructure. The large number of submissions to those benchmarks indicates a very strong interest in these foundational processes by the international community. In part, this also reflects a general philosophy of using measurement-validated models of melt-pool-level processes to understand the location-specific properties at the part level. Although AMB2018-01 received less attention than AMB2018-02, there were still a large number of blind submissions (15).

Looking first at the submissions for AMB2018-02, the largest interest was in modeling the steady-state geometry of the melt pool as a function of laser power and speed. Fully 22% of all 2018 AM-Bench submissions were submitted to this single challenge, again emphasizing the idea that an understanding of the underlying melt-pool physics is considered critical for reliable modeling of complete parts. All the other challenge problems for AMB2018-02 received four to six submissions except for the solidification microstructure challenge that received just one. One possible reason for the relative paucity of submissions for this challenge is that a reliable heat transfer model is required to determine the rate of solidification and temperature gradients that can be used to explore the character and length scale of the solidification microstructure. Successful melt-pool models are a prerequisite for such determinations. It is worth mentioning that knowledge of the elemental segregation within the solidification microstructure is critical for determining the phase evolution that occurs during subsequent heat treatments [6, 8,9,10].

Within AMB2018-01, most of the submissions predicted residual stresses within the as-built part and the deflection that occurs when most of the part is separated from the base. Such predictions are critical for producing reliable components, and this is an area of highly active research. Just a few representative papers on this topic are [20,21,22,23,24,25]. Attempts to predict the phases within the part (1) and the location-specific microstructure (2) were less well represented, likely due to the difficulty in making simulations that bridge the length and time scales between the laser–material interaction and the final evolved microstructure within a completed part. Other difficulties include the lack of relevant and reliable thermo-physical and thermo-mechanical data, and a lack of validated melt-pool and solidification microstructure models.

Comparisons Between Measurements and Blind Modeling Predictions

Another useful factor to examine in Fig. 5 is which submissions were successful and which ones were not, as indicated by the text color of the challenge names. This information serves as a snapshot of how well existing modeling capabilities can predict, a priori, important features of AM builds.

The success criteria for each of the AM-Bench 2018 challenge problems were determined jointly by the corresponding measurement team members and the AM-Bench 2018 Organizing Committee, after the simulation results were received. Since this was the first iteration of AM-Bench measurements and challenge problems, there was considerable uncertainty over how well any of the AM simulations would match the measurements for the various challenge problems. Setting extremely tight, a priori, acceptance criteria that no one could meet would be counterproductive to the goal of distinguishing between different modeling approaches. Another factor that needed to be considered was that some challenge problems were more amenable to quantitative criteria than others. For example, deflection of the AMB-2018-01 specimen after partial cutting from the build plate could be plotted on a single graph and a root-mean-square criterion could be used to evaluate submissions. On the other extreme, one-to-one comparisons between the measured and simulated microstructure of single laser track cross sections for AMB-2018-02 were not possible since the detailed microstructure depends upon the specific configuration of grains that form the boundary of the solidifying melt pool. In principle, statistical criteria could be used, but sufficient measurement data for such criteria were lacking. Therefore, this challenge was judged by visual inspection of the grain sizes, aspect ratios, directionality, and epitaxial character of the submitted results. Finally, the AM-Bench Organizing Committee understood from the beginning that some of the challenges were more “challenging” than others. For example, one possible approach for simulating the phase evolution within thick and thin legs of bulk metal AM builds during isothermal annealing requires a combination of a laser-sample heat flux model, thermal models of the builds of the legs, thermo-kinetic modeling of the location-specific elemental segregation and phase evolution, followed by additional thermo-kinetic modeling of phase evolution during a subsequent heat treatment of this heterogenous structure. Submissions for this challenge were deemed successful if the correct phases and trends were predicted, even if the specific shapes of the precipitates and the time scales of the phase evolution were not accurate.

In Fig. 5, green indicates that at least one of the submissions for that challenge met the success criteria set by the AM-Bench Organizing Committee and the researchers who conducted the corresponding measurements. Red indicates that none of the submissions met those criteria. For all but two of the challenge problems with successful submissions, just one submitted simulation met the success criteria. The first exception is challenge AMB2018-01-RS, residual strains within the bridge structure shown in Fig. 1. Two submissions to this challenge were deemed good and received first-place awards. The other exception is AMB2018-02-GS, the grain structure of single laser track cross sections. Here, one submission received a first-place award and another received a second-place award. The research teams that received awards for the AM-Bench challenge problems may be found on the AM-Bench web page at www.nist.gov/ambench/awards.

Ideally, it should be possible to correlate the quality of the model outputs with the methodology used for the simulations. Unfortunately, the necessary data for such a study were not available since the AM-Bench Organizing Committee did not require submitting organizations to share details about the modeling methods used. Requiring such details would have limited submissions from organizations developing proprietary software. Some of the relevant modeling details are available in other papers in this special AM-Bench issue of IMMI.

AMB2018-01

For AMB2018-01, one partial success story was the prediction of residual elastic strains within the as-built IN625 part. These strains and stresses were measured using three complementary methods: neutron diffraction, synchrotron X-ray diffraction, and the contour method. A complete description of these measurements for the IN625 specimen is published in this issue [13]. Figure 6 shows the complex, scan path-dependent residual strains along the Z axis (perpendicular to the build plate) measured using synchrotron X-ray diffraction [13]. Of the six blind submissions for the residual strain challenge, two stood out by qualitatively matching all of the observed spatially varying strains. One of these teams has a paper in this special issue [26]. For the winning teams, quantitative comparison between submitted simulation predictions and the measurement results shows that the accuracy of the predicted strains is not uniform throughout the part. For some strain directions and regions of the specimen, deviations of 100% are not uncommon. It is worth noting that modeling residual strains within LPBF-built parts requires methods for incorporating the scan path effects at the melt-pool length scale. These benchmark measurements will allow the effectiveness of different residual strain approximation methods to be quantitatively tested. For these first blind submissions, qualitatively correct predictions were the stand-out award winners.

Fig. 6
figure 6

Residual elastic strains along the Z axis within the mid-plane of an as-built IN625 AM-Bench test specimen. The color-coded strain values are in units of micro-strain

Full size image

The partial successes in predicting residual strains did not carry over to successes in predicting the distortion that occurs when the part is partially cut from the build plate [13]. Figure 7 shows the measured and simulated displacements of the top of an IN625 specimen after partial removal from the build plate using EDM. The simulations are the blind submissions to AMB2018-01-PD. Half of the simulations overpredicted the deflection, and the other half underpredicted the deflection by roughly the same amount. None of the simulations matched the measurements well enough to meet the success criteria. Since accurate deflection predictions typically require accurate residual strain predictions, it is not surprising that the qualitatively correct strain simulations described above failed to accurately predict the deflection results.

Fig. 7
figure 7

Measured and simulated deflections of an AM-Bench test specimen following partial removal from the build plate. The black curve presents the measurement results, while the other curves each represent blind model submissions. The measurement uncertainties are much smaller than the symbols

Full size image

For the remaining three AMB2018-01 challenges, only a few submissions were received, and only partial results were submitted. This should not be surprising since predicting the solidification microstructure within macroscopic AM builds is extremely challenging. For example, the winning submission for the microstructure challenge performed comprehensive simulations using multiple, complementary methods. The CALPHAD approach was first used to examine the expected phase diagram and to determine the liquidus and solidus temperatures. Next, finite element modeling was used to extract cooling rates, critical velocities, and thermal profiles as a function of position within a thin leg. Finally, empirical methods were then used to determine the character of the solidification microstructure and the expected length scale. The same group also submitted preliminary phase field solidification models. The only submission to the phase evolution challenge also used the CALPHAD approach to simulate the formation and evolution of δ-phase precipitates during 800 °C annealing of the AM-build IN625 specimen.

AMB2018-02

For AMB2018-02, a discrepancy between the AMMT and CBM measurements was found after the experiments were completed, which stemmed from erroneous laser power and/or laser spot size measurements. All of these issues have since been fully reconciled [16]. Despite the lateness of the revelation, ten modelers submitted melt-pool geometry predictions (AMB2018-02-MP) in comparison with the CBM measurements, AMMT measurements, or both. For this challenge, predictions of the melt-pool length, depth, and width were required. Although this challenge received more submissions than any other, none of the ten submitted melt-pool geometry simulations adequately matched the measured results. For example, Fig. 8 shows comparison between the measured melt-pool length for the CBM and AMMT systems and the simulated results. Here, the colors represent different challenge submissions and the lengths of the bars give the percentage difference from the measured length. Although some modelers were able to predict one or two of the dimensions well, no single model stood out as predicting all three melt-pool dimensions accurately for either the CBM or AMMT measurements.

Fig. 8
figure 8

Comparison between submitted simulated melt-pool lengths and the measured values. The colors represent different challenge submissions

Full size image

Similarly, the predicted surface topography for the melt traces (AMB2018-02-TP) differed significantly from the measurements obtained using confocal scanning laser microscopy [16]. As shown in Fig. 9a, deviations of over 200% were common. Here, for example, A-Max indicates the maximum height of the simulated laser track for case A. Although the magnitudes of the heights did not match the measurements, most of the five submitting groups did predict the measured general shape of the tracks. Figure 9b shows the position-averaged shapes of the measured AMMT profiles. The points in the transverse profiles shown in Fig. 9b are the averages determined for each location from nine transverse profiles taken with a spacing of 1 mm over the middle 8 mm of the 16 mm long tracks. As will be discussed in the topology measurements paper included in this issue, the topology of the tracks varied considerably even in this middle section where propagation should be as close to steady state as can be achieved. [17] Clearly, turbulent processes are occurring with a number of stochastic contributions influencing topology. It appears that this variability will need to be accounted for if models are to be used to predict the morphology of builds and the probability of different types of build flaws.

Fig. 9
figure 9

Comparison between the five laser track predictions and the average for the minimum and maximum height determined from nine transverse profiles (a) and the average determined for each position in the profiles (b). The colors represent different challenge submissions

Full size image

Perhaps surprisingly, some of the predictions of cooling rate after solidification, grain structure, and solidification microstructure [27] closely matched the corresponding measurements [14], even though the melt-pool geometry predictions did not. Several factors likely contributed to this agreement, all of which bode well for the possibility of developing predictive models for microstructure and material property evolution. First, the cooling rate immediately following solidification clearly depends largely upon the rate at which heat is conducted through the sample away from the top surface of the solidified melt pool. Although this rate must be affected by the detailed temperature distribution left by the original melt pool, this effect appears to be small compared to the overall thermal conduction. Second, the solidification microstructure depends upon the solidification thermal gradient, G, and the solid–liquid interface velocity, V. Basically, G/V affects the morphology of the solidification structure and GV (the cooling rate) affects the scale of the microstructure [28]. Since order-of-magnitude changes in G and V are typically required to significantly alter the solidification microstructure, the simulation results are sufficiently accurate to correctly predict the solidification microstructure. Third, the successful grain structure predictions used a cellular automata model with epitaxial growth. Nucleation rates were assumed to be small enough to be neglectable. For IN625 single laser traces, this assumption is reasonable, and the resulting simulations strongly resembled the grain structures visible in the electron backscatter diffraction (EBSD) images [14]. For materials and geometries where grain nucleation plays a significant role, incorporation of a reasonable nucleation rate would be required. One final remark about the submissions for the AMB2018-02 challenges is that a wide range of predictions was received, indicating that several different approaches were being evaluated. In general, it was found that those simulations that included the most rigorous underlying physics produced the best results.

AM Community Input from the AM-Bench 2018 Discussion Session

At the end of the AM-Bench conference, a discussion session was held to summarize lessons learned from the 2018 test series and stimulate ideas for future benchmark tests. The session was divided into two sections with discussion panels selected by the Organizing Committee, and questions to encourage discussion among the participants of the conference. Overall, comments on the quality and success of the benchmark measurements and conference were very positive and, as with the model submissions, most of the discussion focused on modeling for metal AM as opposed to polymer AM. Please note that the following is a summary of the comments made by conference participants during the discussion session and does not necessarily reflect the views of the authors or the AM-Bench Organizing Committee.

Public Discussion

Discussion in the first session focused on a general understanding and purpose of modeling for AM, current limitations of AM models, and the requirements for accuracy, speed, and uncertainty. In general, modeling should enable informed design of the part and build process, with a goal of reducing the time and expense of producing qualified parts. Modeling can be used to develop a better understanding of aspects that cannot be measured and can reduce the required number of experiments through integrated computational design of experiments, exploration of process space, and a focus on optimizing multiple parameters simultaneously. Modeling can also be used as a synthesis task to hasten development of reference guides (e.g., references similar to Ashby diagrams for material selection). Moving forward, however, it must be determined how modeling can affect the standardization effort. Without that, it was suggested, industry cannot use modeling in a practical manner.

The audience also expressed the limitation of modeling due to the time required to develop models for AM. Currently, it is common for an entire doctoral study to be dedicated to understanding a single geometry, material, and machine or process combination. Changing these factors requires more work than is practical for industrial use. Surrogate modeling may address this through increasing the speed of available models, but there should be a priority on how to transfer knowledge as process variables change. This led to discussions of accuracy as it is not clear what level of accuracy would be required of a “fast” model. Industry participants added that currently they are looking more at trends, but accuracy will be required for distortion predictions, suggesting that the level of accuracy will be highly dependent on the model output in question. [The authors note that these remarks are consistent with the discussion of the blind challenge submissions for residual strain and part distortion in the previous section.] This also led to discussions on the inclusion of uncertainty statements (e.g., error bars) with simulation results, which can be difficult given the range of sources of uncertainty but will allow for better comparison of model results. Furthermore, it is a goal of the AM-Bench committees to target the greatest sources of uncertainty for improvements which will require input from the modeling community on model sensitivity so that the committee can properly target the measurements that require improvement.

The second session focused on the future of AM-Bench. With the aspects from the first session in mind, the audience suggested continued advancement of modeling residual stress and microstructure, but also to include post-process treatments (e.g., shot peening or heat treatment) and history of variables to investigate the evolution of a part’s microstructure from the build process through to the final part. Investigations of thermal buckling as well as reducing the “thin” wall sizes down to 200 μm were also suggested. Furthermore, some of the challenges may have been too complicated to address and perhaps addressing a new alloy would be a better use of the benchmark instead of increasing complexity. Simplified alloys (binary or tertiary) could also reduce complexity and could help with intellectual property and export restrictions that participants faced.

Discussion of measurements to improve the accuracy of modeling results included beam diameter, uncertainty of process parameters and feedstock properties/condition, thermal conductivity within different parts of the build (i.e., powder, liquid, solid), and porosity. It was noted that there is also potential to interrogate mechanisms that affect the physics of the process, such as in situ diffraction for nucleation events. Finally, it was stressed during the discussion, and certainly worth reiterating to the readers, that the community should not be constrained by what is thought to be possible in terms of measurement. If the need is present, there are creative and talented people in the research community capable of determining new methods of measurement to meet those needs as the AM-Bench series continues.

AM-Bench Data Curation

The data associated with this effort have been curated using the configurable data curation system (CDCS) software [29, 30] as deployed in a customized instance at www.nist.gov/ambench/benchmark-test-data. The CDCS software uses Extensible Markup Language (XML) formats to describe the metadata associated with the data, and large data files (e.g., images) are stored as binary large objects (BLOBs) separately in MongoDB databases within the CDCS framework. A custom XML schema was developed to curate the data and is available at schemas.nist.gov [31]. The schema is divided into two parts. The first section identifies the specific material, build conditions, and location of the sample from a given AM build process. The second section of the schema provides a list of characterization methods. For each characterization method, a specific set of metadata is outlined. The specific XML schema used is available at the schemas.nist.gov repository. The data can be entered into the repository either using the web interface or using a representational state transfer application programming interface (REST API). In addition, using either the web interface or the REST API, the data can be searched using a keyword search across all the data or complex queries can be developed based on the XML designated fields. Future work includes developing specialized exporters to provide data in specific formats. The 2018 AM-Bench data are still being updated as additional measurements and analyses continue. We expect to complete this process during 2020, at which point incremental changes will cease and a version number will be assigned to the finalized data sets. Subsequent changes will be documented and assigned new version numbers as needed.

Conclusions

Although the long-term impact of AM-Bench 2018 resides in its permanently archived and freely available measurement data, 2018 AM-Bench measurements, measurement challenges, and conference have provided a wealth of additional valuable information about current interests and directions of research aimed at simulating AM processes. It also provides a snapshot of AM simulation capabilities as of May 2018. The primary conclusions reached from analyzing the 2018 AM-Bench challenge submissions are the following:

  • Worldwide modeling efforts for metal AM appear to greatly exceed those for polymer AM (45 submissions vs. 1 submission).

  • There exists substantial interest (30 out of 46 submissions) in modeling fundamental melt-pool processes, presumably to provide a rigorous foundation for simulating part-scale properties and behavior.

  • Overall, simulations that included substantial underlying physics more accurately matched the benchmark measurements than those that used empirical methods.

  • Although it is relatively easy to calibrate a melt-pool thermal model to accurately provide two of the three melt-pool dimensions, getting all three correct is significantly more challenging.

  • Exact simulation of the melt-pool geometry is not required to obtain excellent microstructure simulation results.

  • Simulations of residual elastic strains can produce qualitatively correct results that include laser path effects. However, substantial deviations from the measured strain values existed in all submitted results.

  • All of the submitted simulations for part distortion following EDM cutting deviated substantially from the measurements. Inaccuracies in the predicted residual strains/stresses are the most likely cause.

  • Accurately simulating the location-specific microstructure within a 3D build is clearly beyond the current state of the art, but successes in modeling smaller-scale processes and in coupling such information to larger length scales are encouraging.

  • As alluded to above, some melt-pool microstructure models were successful even though the underlying physics was not fully accounted for. These successes bode well for the eventual development of reduced order models that can run on workstation-level computers and produce useful results for industry in reasonable time frames.

  • Although 60% of the modeling challenges had at least one submission that met the success criteria, other submissions to these same challenges typically made predictions that were much less accurate. The wide range of predictions demonstrates that benchmark measurements can play a crucial role in determining what modeling approaches are most effective for a given AM process.

Valuable lessons were learned in developing and carrying out this first round of benchmark measurements, challenge problems, and conference. The importance of closely engaging the international community is crucial, as is providing a comprehensive, extensible database of measurement results that is freely accessible online. The AM-Bench committee members provided crucial input when the first round of measurements was being developed, and their input for the next round of benchmark measurements has already been solicited. The plenary discussion session held during the AM-Bench 2018 conference provided valuable insights into existing AM modeling needs and capabilities, as well as valuable suggestions for future AM-Bench measurements. Future AM-Bench conferences should expand upon these discussion sessions. Finally, based upon lessons learned from AM-Bench 2018, work on overhauling the existing AM-Bench database and Web site is in progress to provide a firm foundation as future iterations of the AM-Bench measurements, challenges, and conferences go forward.