Overview

The Air Force Research Laboratory (AFRL) Additive Manufacturing Modeling Challenge (AMMC) series was devised as a means to assess the Additive Manufacturing (AM) modeling community’s ability to predict, in a locally resolved fashion, details of the structure and subsequent mechanical performance of metallic materials fabricated with a laser powder bed fusion (LPBF) process. The series was originally announced in an article in JOM in March 2018 [1], and the four individual challenge problems are described in several articles submitted to Integrating Materials and Manufacturing Innovation [2,3,4,5,6].

As described in a previous publication [1], challenge problem topics were crafted with the input of community leaders in AM process modeling, performance modeling and automated design methods. Using the advisory panel’s recommendations, AFRL researchers finalized the problem statements, coordinated experiments and produced publicly distributable data packages. The challenge problems that were selected focus on process-to-structure predictions and structure-to-properties predictions at both the macroscopic/aggregate and microscopic/local scale. For the process-to-structure challenges, process history (i.e., scan paths, laser power and speed, layer thicknesses and times) was provided as calibration and input data. A detailed description of the material state /structure was provided for the structure-to-properties challenges. These challenge themes are outlined schematically in Fig. 1.

Fig. 1
figure 1

Organization of AFRL Additive Manufacturing Modeling Challenge (AMMC) Series

Full size image

Challenge Series Implementation

All challenge specimens were produced using an aerospace relevant, commercially available Ar gas atomized nickel-based superalloy IN625 powder. Commercial software was utilized to lay out the builds on a commercial EOS M280 laser powder bed fusion system.

AFRL and its partners performed a series of characterization activities to empirically measure the quantities of interest and then split these data into subsets for calibration and validation activities. For example, the microscale structure-to-properties challenge utilized high-energy X-ray diffraction microscopy data collected at Argonne National Lab [7] to provide elastic strain tensors for individual grains during tensile loading and a 3D description of the sample was reconstructed from electron backscatter diffraction, optical and SEM data collected using serial sectioning techniques developed at AFRL [8]. A data descriptor article for each challenge reviews the experimental procedures and describes the data available [2,3,4,5,6].

A primary goal of this challenge series was to provide sufficient background information and calibration data to attract the broadest participation. Data and documentation were compiled into problem statements consisting of (1) general problem statement, (2) background information, (3) data for model calibration, (4) description of desired predictions, (5) input for challenge questions, (6) challenge question and scoring, and (7) supplemental data [9,10,11,12]. Experimental techniques and the resulting raw and analyzed data were described within the relevant sections of the challenge statement document. The four challenge statements reference an associated data package or dataset that, in general, included the raw and processed data. When applicable, the scripts used for analysis were also included in the dataset. [9,10,11,12]. The challenge question and scoring section of the challenge statement described the challenge specific answer template and the grading rubric. All participants were required to submit responses in a standardized tabular form.

A public website has been developed [13] to host general information about the challenge series including the problem statements, answer templates, and information on how to access the data. The supplemental information for the present article includes a copy of the website for reference. The materials data facility (MDF) [14, 15] hosts the 21 GB of data and supporting documents associated with the challenge datasets. To ensure data longevity, the data are publicly available for mirroring, and MDF guarantees the data availability through 2028. The MDF platform allows users to browse, search, and download specific files or the whole dataset using standard HTTPS protocol or Globus interfaces. To gain access to the challenge datasets, interested parties use a free Globus account and join a user group (AFRL AM Modeling Challenge Series) that allows AFRL researchers to communicate with the signed up participants.

The details of the challenge questions were accessible starting in November 2019 [16] and respondents were required to submit answers by February 2020. From November to February, respondents were encouraged to submit technical and logistical questions, with all questions and the answers being made available on the public website and archived in the supplemental information for this article. Respondents were informed of their absolute score and ranking relative to other respondents in March 2020, and top performers were publicly announced July 2020 [17]. A workshop to debrief the participants as well as to discuss the outcomes and opportunities to improve both the modeling and experimental procedures was held virtually in September 2020 and hosted by the National Center for Defense Manufacturing and Machining (NCDMM).

Challenge Series Results

Across the four challenges, 20 responses from 16 unique teams were submitted. Respondents included large and small commercial software developers, academic institutions and national labs. Several responses were a collaborative effort across multiple institutes. The respondents with the highest score in each challenge received public recognition as top performers at the September 2020 workshop. The following summarizes the number of submissions and the top performer for each challenge.

Challenge 1 Macro-scale Process-to-Structure Predictions Challenge: Given processing details, predict macroscale residual strain.

Total Submissions: 6

Top Performer: Dassault Systems Government Solutions Corp.

Challenge 2 Micro-scale Process-to-Structure Predictions Challenge: Given processing details, predict deposit geometry details at the single bead, single layer scale.

Total Submissions: 5

Top Performer: The Wing Kam Liu Group, Northwestern University.

Challenge 3 Macro-scale Structure-to-Properties Predictions Challenge: Given representative microstructure characteristics, predict aggregate stress–strain behavior.

Total Submissions: 5

Top Performer: QuesTek Innovations LLC.

Challenge 4 Micro-scale Structure-to-Properties Predictions Challenge: Given an explicit microstructure representation, predict elastic strain behavior in particular grains.

Total Submissions: 4

Top Performer: University of Utah, Carnegie Mellon University and Los Alamos National Laboratory (Carter Cocke, Dr. A.D. (Tony) Rollett, Dr. Ashley Spear, Dr. Ricardo Lebensohn).

Challenge Series Workshop

NCDMM hosted the virtual workshop 14–17 September 2020 to debrief challenge results. An invitation to attend and present at the workshop was extended to all respondents in the challenge series. Invitations were also extended to Department of Defense (DoD) and National Laboratories, and DoD aerospace supply chain contractors (material suppliers, subcomponent producers, and original equipment manufacturers).

Each of the four half-day sessions focused on a specific challenge and included an AFRL presentation, respondent presentations, award announcements and discussion periods. AFRL area experts presented an overview of each challenge question and a review of the anonymized submission results. Of the 20 total respondents, 11 opted to present at the workshop and focused on their approach and the specific techniques utilized to make predictions. A two-hour discussion period at the end of each day was utilized to field general questions and collect feedback from the respondents and workshop attendees.

Lessons Learned

The workshop and participant feedback provided AFRL with valuable insight into how to improve future modeling challenges. A major opportunity for improvement focused on communication between the experimentalists and the modelers. The modeling community voiced a need for clear and simple definitions and descriptions of data. Clarity and consistency in the descriptions of the challenges are critical. For instance, in challenge 1, the “elastic strain tensor” was defined as the prediction target early in the problem statement, but in other locations the ambiguous term “strain tensor”’ was used, leading to some confusion.

One mechanism to catch such errors was the public Q&A portion of the website. We received and responded to 18 technical questions across the four challenges before submissions were due in February 2020. Although the written Q&A section was necessary, it was not sufficient.

One proposed way to improve communication is to hold informational sessions at or before the release of modeling challenges. In addition to reviewing the logistics, problem statements and data, the experimentalists should be on hand to explain the techniques and field questions. Additional ‘office hours’ will give modelers additional opportunities to interact directly with experimentalist and challenge coordinators as questions arrive. Recording and posting these sessions will ensure all participants have access.

By developing and executing this public modeling challenge series, AFRL was able to assess participants’ ability to predict AM properties of interest given a specific set of input data. The longer term, more impactful outcome of the challenge series is the curated, documented and publically available data used in each of the four challenge series.