Abstract
Design has been expanding its position. The design has been expanding its position. because, the development of technology, to grow demand that solving problems using design methodologies [9, 13]. As a result, the capacity of individual designers is becoming more important, and the importance of ‘portfolios’ expressing and evaluating their capabilities is increasing. In the industrial society, the portfolio is used as an indicator of competence of designers, but the evaluation method and criteria depend on the subjective view of the evaluator so that the competence of the designer is not objective. Although both designers and evaluators agree on this problem, research on the portfolio itself, as well as the portfolio evaluation area, is very limited [3]. Therefore, this study is based on the priority of the components that should be considered in the evaluation of the completeness of the portfolio by analyzing the components of the portfolio so that the portfolio can be evaluated as an index for evaluating the individual competence of the designer Evaluation guidelines were presented. In order to do this, we conduct surveys and in-depth interviews with designers engaged in the business to understand the needs of the portfolio, and analyze the component data of the portfolio extracted based on the analysis, and prioritize the portfolio components that should be considered for completeness determination This study has significance in that it is presented through data analysis.
You have full access to this open access chapter, Download conference paper PDF
Similar content being viewed by others
Keywords
1 Introduction
With the recent developments of AI (Artificial Intelligence) technology have caused various changes in various industrial fields [13, 17]. Also, the design industry is also being developed and expanded through various changes [10], and the role of design in the industrial society is increasingly recognized as an important means. Since the design industry is an industry on which human resources are based, one’s personal competence is more important than other industries, and the need for competent designers is increasing. Therefore, the completeness of the portfolio, which serves as an indicator for evaluating the personal competence of the designer and as an individual career certificate, is becoming more important. As such, we are aware of the importance of the portfolio, but the evaluation criteria depend on the subjective opinion of the evaluator. Therefore, not only the designer but also the evaluator is confused about producing and evaluating the portfolio.
The purpose of this study is to find out the objective index for evaluating the personal competence of the designer as a preliminary study for the AI - based service development that objectively and quantitatively evaluates the portfolio, identifies priorities of components that affect portfolio completeness assessment and provide evaluation guidelines. In order to accomplish the purpose of this study, we have identified the needs of designers and portfolio evaluators who produce portfolios through in-depth interviews and in-depth interviews. Based on this, Python was used to extract and analyze the 820 component data of the portfolio, and the priorities of components to be considered in the portfolio identification were derived. This study is meaningful in that a new direction using data was devised and presented for objectification and quantification of the portfolio evaluation method. As a result of this study, it is expected that designers will provide a guideline to express their competence more effectively in the portfolio and more objective and standardized evaluation criteria to the evaluator. In addition, if subsequent research based on this research is continued, it is expected that AI services for objective evaluation can be constructed in the evaluation of the completeness of the portfolio.
2 Literature Review
2.1 Portfolio and Evaluation Methods
Design is a specialized area for presenting solutions to problems and acting on those tasks. Therefore, compared to other direct groups, designers are each valued for their individual competencies. because the portfolio is the most important material for assessing the competency of individual designers [6]. A portfolio is a communication tool that shows the designer’s personal competence and communicates with others as a mirror showing the designer’s thoughts and problem-solving methods, rather than simply a completed collection of works. (Kim 2008) was carefully analyzed according to the characteristics of the structural elements of the interior design portfolio and arranged the elements of the portfolio to focus on the way the presentation of the portfolio. (Han [6]) was presented in six categories of guidelines for the objective competency evaluation of designers through portfolios through expert in-depth interviews, looking at portfolios from the perspective of designer recruiters. This study is significant that this is a study that objectively presents the evaluation criteria of the designer through the portfolio. However, the limitations of the study mentioned by the authors of this paper are as follows. 1. Participants in the in-depth interview for the elicitation of needs were limited. All the interviewee were evaluators. 2. Since the evaluation method of the portfolio has already been evaluated as subjective and abstract language, it is difficult to objectify the evaluation method. The guidelines are shown in Table 1.
2.2 Changes in the Industrial Sector Due to the Emergence of AI
In addition to recent advances in AI technology, many changes have been taking place throughout the industrial society of Automobiles, robotics, education, etc. In the design area, you can find many examples of using AI. Companies such as Adobe, Microsoft, and Autodesk that produce 3D Tools and 2D Tools that are used primarily by designers provide artificial intelligence-based ‘intelligent services’ to provide designers with the convenience of using tools. Services such as Google’s Quick Draw is to convert complete rough sketches to detailed graphics [12]. And such as ‘tailorbrands’, ‘hatchful’ and ‘Logoshi’ are automatically to create logos. The application of artificial intelligence in the design area has been applied to a variety of areas, from design tools utilized by professional designers to services that make it easier for non-professionals to access the design.
3 Preliminary Survey
A questionnaire survey was conducted on 117 designers. The purpose of this questionnaire is as follow. (1) to understand how designers perceive portfolios. (2) To find out the needs for an objective portfolio evaluation method (Table 2).
A total of 117 people participated in the survey, and 2018.09.26–2019.09.30. It took 5 days in total. Data collected from questionnaires were analyzed using the SPSS program, and frequency analysis was conducted. The general characteristics of the questionnaire are shown in Table 3.
Design skills and design expression (65.8%), design thinking (60.7%), and planning ability (57.3%) were the most frequent factors in the frequency analysis. Design Tool skill & design expression was remarkably high. The creativity of designers was ranked 5th with 31.6% (Table 4).
Portfolios, academic backgrounds, and self-introduction were the first, second, and third most important factors in evaluating designer competencies. Among the factors that assess the competence of designers, the portfolio was significantly higher at 98.3%. As a result, it can be seen that designers are aware of ‘portfolio’ as a tool for expressing their competence. Table 4 also shows that the ‘portfolio’ is effective in evaluating ‘Design Tool & Design Expression’, which is an essential capability for designers.
In order to achieve the purpose of this first study, we conducted post interviews on the services needed to develop the competency of the designer. This is because interviewing methods are more effective in identifying the needs of designers. The interview results were coded and the needs were typed as shown in Table 6. The frequency analysis showed that ‘Education service (40%)’, ‘Portfolio service (24%)’ and ‘Community service (20%)’ were ranked first, second and third respectively. Designers want to grow. Thus the need for education services so high. The need for portfolio services was also high. Portfolio service needs were detailed in the order of ‘Production Guide’, ‘Attachment’, and ‘Objective Evaluation’.
This study confirms that the importance of portfolio and objectification of the evaluation method are needed in the industrial structure where a designer’s personal competence becomes increasingly important through previous research and pilot questionnaire. Confirms the priority of components affecting the evaluation of portfolio completeness, suggest evaluation guidelines.
4 Method
4.1 RandomForest
Random forest used in this study is to generate several decision trees from data and then to predict the most selected class among the predicted values of each individual tree [1, 16]. In other words, it is an ensemble technique that improves learning performance by applying several simultaneously. The ensemble means harmony or unification in French, and it learns several models in machine learning and predicts better values than one model using the predictions of the models. This method is called an ensemble learning or an ensemble method [14, 16]. Random Forest works by constructing more than 30 datasets from the same data, extracting them from the same data, applying decision trees to each, and collecting the learning results [5]. However, each tree only learns a few of the variables, which is an attempt to increase diversity by making the individual trees view the data differently [14]. Random forest is a way to eliminate the overfitting problem of the training data of existing decision trees. To realize this, it is necessary to make many decision trees [15, 16]. Each tree should be well predicted and distinguished from other trees [14, 16]. Random forests are generated randomly when a tree is created so that the trees are different from each other. There are two ways of randomly choosing the data to be used when creating the tree, and randomly selecting the characteristics in the partitioning test [14, 16].
4.2 Research Procedure
Design portfolio is a critical factor in evaluating the capability of a designer. However, there is no standard way to evaluate a design portfolio. Additionally, there has been no academic study to give a guideline for evaluating a design portfolio. We had an in-depth interview with designers in the field to collect possible criteria for design portfolio evaluation. According to the criteria, we organized the elements that good design portfolios have and made a data-driven classifier for identifying good design portfolios. Eventually, we came up with a guideline for evaluating a design portfolio based on our study. This guideline suggests the priority of the elements that organize good design portfolios.
Step1
Aleading study of portfolio components for portfolio improvement and for the development of business designers’ needs (Kim 2008) and the prior art of the evaluation of the Portfolio Assessment guidelines (Han 2011) was, based on this, the required capabilities of the designer were typed into five. Based on the structure of the Korean Design Industry (Korea Industrial Design Statistics Report, Kidp 2017), Survey and In-depth interviews have conducted with 15 designers who were in the office of start-up, Design Agency, and in-house. As a result, it was confirmed that the main evaluation criteria for evaluating the portfolio are different for each group (Table 7). In this study, tried to show between the Designers competency defined by ‘basic skills’ and portfolio elements are related. Table 8 shows the types of designer competencies evaluated through the portfolio through the interview coding process (Table 8).
Step2. Portfolio Gathering
Behance (behance.net) is an online website that can display and search works created by designers from around the world with Adobe Creative Cloud [2]. Pinterest (pinterest.com) is a social network service that users post and share images with other users. In this study, Using Python, the 900 Portfolios, which is open online, such as ‘Behance’ (behance.net) and ‘ Pinterest’ (pinterest.com), has been Crawled [2, 11]. Among them, the 820 Portfolios were used in the analysis for convenience in the analysis, the 80 Portfolios which are non-portfolio images and dynamic images (such as a GIF) were excepted. The components of the portfolio are largely divided into images and texts, and each itemized component of the data extracted from the 820 portfolios is shown in Table 9.
Step3. Data Preprocessing
The 820 portfolios are to be used for analysis, 40 were divided as Good and 780 were divided as Normal. Standard of between good and normal has been divided as of whether or not having a ‘Behance recommendation label’ for the convenience of analysis, the extracted data is encoded with matched with numbers and then, the learning model could process them. The N-gif and N-tool were not encoded because their values are already represented by integers and the values range from 0 to 5. On the other hand, main-color and key-color are encoded the extracted RGB values based on the color tone. A total of five color tones were used as the standard, monotone as 1, pastel tone as 2, vivid tone as 3, deep tone as 4, and natural tone as 5. The value of N-Text is encoded by 200 units, with 0–199 as 1, 200–399 as 2, 400–599 as 3, 600–799 as 4, and 800 or more as 5 (Table 11). We select the portfolio completeness evaluation as the ‘target variable’ and the six attributes selected in Step 3 as the ‘predictor variables’. The variables used for the prediction of the portfolio completeness among the total variables were selected through the preliminary study and the in-depth interview. A total of six variables were selected, one for design skill, one for comprehension and analysis, and one for originality. Selected variables were extracted from 820 portfolios and created a dataset (Table 10).
Step4. Data Partition
In order to make an ideal model using data mining, it is desirable to create multiple prediction models from one data set and compare and analyze them [7]. Therefore, we divide the whole data into analytical data and verification data, create a model with analytical data, and apply the verification data to the model to compare the performance of the model. In this study, was divided by 7:3. Since random forests arbitrarily select variables to construct a tree, the ratio of analysis data to verification data does not affect the analysis [16].
Step5. Data Training and Evaluation
Random forests can be used to rank the importance of each variable in a classification or regression problem [1, 16]. Feature_importances_ was used to calculate the importance value of predictive variables. The data used in the analysis are unbalanced in the target variable. In the case of unbalanced data in RandomForest, there is a way to compensate overfitting by assigning a weight to a small number of variables [4]. However, in this case, there is a drawback that it is more susceptible to noise, and analysis by a weighted method does not affect the performance of the prediction model [4]. To build a RandomForest model, you must define the number of trees to be created. You can set the number of trees by the n_estimator value of RandomForestClassifier. In this analysis, n_estimator = 500, random_state = 42 was set, and a training model was created using scikit-learn. In order to reduce data noise and reduce the probability of overfitting, we normalized the normalized score to 0 with a mean and a standard deviation of 1. Principal Component Analysis (PCA) was used to build the model.
5 Result
5.1 Feature Importance
RandomForest was used to derive important feature importance for class discrimination and feature priority. As a result, N-Tool, N-gif, and N-Text were ranked first, second and third. This is a significant result because it reflects the latest trend in which the portfolio using dynamic images such as gif and video has appeared. Also, according to a result of in interviews has conducted with 15 designers, there were many opinions that text is important in portfolio evaluation. In other words, the portfolio is recognized as a means of communicating between the designer and the evaluator. so Text is considered more important than Image in the portfolio because the text implies that the value of the designer and the process of solving the idea. Although this study identifies only the number of simple texts, there are limitations, but it is meaningful in that it is in line with the main elements derived through in-depth interviews. The feature priorities for class discrimination are summarized in Table 11 below.
5.2 Accuracy Score
Accuracy Score to create an ideal model, data is divided the data set into two parts: ‘training data’ and ‘test data’ at a ratio of 3:7 and analysis using RandomForest. As a result, as shown in Tables 12 and 13, the training score is 0.9913 and the test score is 0.959933 (Tables 12 and 13).
6 Conclusion and Suggestion
Portfolios are a fundamental document for assessing the competence of designers, and the importance of the portfolio is rapidly increasing. However, related research is insufficient. In addition, existing research is only a conceptual guideline based on the opinion of the evaluator. Therefore, this study analyzed the literature related to the portfolio evaluation, analyzed and typified the portfolio components through in-depth interview with the working designer, and presented a comprehensive direction considering the specificity of the Korean design industry structure. Based on the results of this study, it is expected that designers will use the dynamic image to create a portfolio that reflects the latest trends in the portfolio and will help improve the expression and evaluation of individuals when considering the weighting of the text in portfolio formulas.
However, this study has the following limitations. the number of portfolio samples used in this study is 820, which is not enough. In the process of creating a dataset as a property of the portfolio, class classification criterion is ambiguous, it is classified as Good & Normal. When evaluated the portfolio through in-depth interviews, confirmed that the main competence such as designer ‘s understanding and analytical ability was evaluated through the context of the text. but, in this study, attribute describing the text context in the portfolio could not be found.
References
Amit, Y., Geman, D.: Shape quantization and recognition with randomized trees. Neural Comput. 9(7), 1545–1588 (1997)
Behance. https://www.behance.net/. Accessed 15 Dec 2018
Blackburn, T.: Korea Design Statistical DATA, pp. 48–58, 223–242. Korea, Seoul (2017)
Chen, C., Liaw., Breiman, L.: Using random forest to learn imbalanced data. Technical report, University of California, Berkeley, US (2014)
Guido, S., Mueller, A.C.: Introduction to Machine Learning with Python. O’Reilly Media, Sebastopol (2016)
Han, S.M.: A study on the objective capability evaluation method through a designer’s portfolio -focusing on visual designer-, Sungkyunkwan University, Korea (2011)
Jeong, H.K.: Development of a model for preventing informally discharged cancer patients using medical records in a university hospital. Unpublished master’s thesis, Chungnam National University, Daejeon (2004)
Kim, H. J.: A study on the characteristic contents structure in the interior design portfolio, Sookmyung Women’s University, Korea (2008)
Kim, J.K., Kim, H.H.: It’s Design, Communication books, Korea (2011)
Kim, D.Y.: The changes in design area over time: focusing on the changes in the design department name and design profession field, Hong Ik University, Korea (2017)
Pinterest. https://www.pinterest.co.kr/. Accessed 28 Dec 2018
Quick Draw. https://quickdraw.withgoogle.com/, Accessed 20 Feb 2019
Shin, D.J., Kim, H.H.: Some thoughts about expansion of the design area according to change of the times – focused on invisible area. J. Digit. Des. 13(3), 321–329 (2013)
Vanderplas, J.: Python Data Science Handbook : Essential Tools for Working with Data. O’Reilly Media, Sebastopol (2016)
Wikipedia. https://en.wikipedia.org/wiki/Decision_tree. Accessed 20 Feb 2019
Wikipedia. https://en.wikipedia.org/wiki/Random_forest. Accessed 20 Feb 2019
World Economic Forum: The Future of Jobs: Employment, Skills and Workforce Strategy for the Fourth Industrial Revolution. World Economic Forum, US (2016)
Acknowledgment
This work has been conducted with the support of the “Design Engineering Postgraduate Schools (N0001436)” program, an R&D project initiated by the Ministry of Trade, Industry and Energy of the Republic of Korea.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Ryu, J.M., Lee, K. (2019). Guidelines for Evaluating the Completeness of the Portfolio. In: Stephanidis, C., Antona, M. (eds) HCI International 2019 – Late Breaking Posters. HCII 2019. Communications in Computer and Information Science, vol 1088. Springer, Cham. https://doi.org/10.1007/978-3-030-30712-7_7
Download citation
DOI: https://doi.org/10.1007/978-3-030-30712-7_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-30711-0
Online ISBN: 978-3-030-30712-7
eBook Packages: Computer ScienceComputer Science (R0)