Physicians are faced with unique responsibilities and challenges that sometimes demand interpersonal skills and abilities that are not part of the traditional graduate medical education curriculum [1]. In particular, physicians in leadership roles are often ill-equipped to deal with the relational demands of leadership as they are often promoted based on excellence in their clinical and research practices [2]. This lack of leadership acumen was borne out in a 1999 national survey of physician-leaders which indicated that the non-clinical skills that were most in need of professional development were: oral communications, listening ability, team building and being a team player, conflict resolution, and general interpersonal skills [3]. Furthermore, the heavy demands of administrative roles may become even more complicated for physicians when the setting changes to a professional medical association (PMA) (e.g., American Medical Association, American College of Surgeons, etc.). Leadership roles in professional medical societies may involve overseeing peers with similar training, education, skills, and sometimes greater clinical leadership experience. For instance, it is possible for a physician to be President of a PMA that includes his/her Chair or Chief Medical Officer from his/her home academic medical center or community hospital. These types of dynamic relationships and interaction may require a level of emotional intelligence (EQ) that most physicians are typically not able to achieve [4].

Prior research has demonstrated that the key competencies for physician-leaders include interpersonal skills and particularly emotional intelligence [4]. Emotional intelligence refers to “effectively understanding oneself and others, relating well to people, and adapting to and coping with the immediate surroundings to be more successful in dealing with environmental demands” [5]. Some researchers argue that the training provided by degree and certificate granting programs is insufficient and that healthcare institutions must develop their own intramural programs to cultivate physician-leaders [4]. We could—and perhaps should—extend this recommendation to include professional medical societies who want to foster effective leadership within their physician constituents.

At the root of this issue may be the bias in medical education curriculum that is devoted to professionalism and ethics to focus almost exclusively on patient interaction and neglect the significance of interaction with colleagues and healthcare staff to workplace success [6]. Furthermore, continuing education programs on professionalism and interpersonal/communication skills in the workplace are scarce for physicians [7] despite the efforts of the Joint Commission on instituting standards relating to disruptive physicians. Given the dearth of continuing medical education on professionalism and interpersonal/communication skills, physicians may have to utilize non-medical, and potentially less credible, training to garner helpful information in this important performance area.

The use of 360-degree surveys or multisource feedback has shown promise as an effective tool for initiating growth and development of interpersonal skills and leadership acumen [8–10]. Research on 360-degree feedback of managers has consistently shown that others’ ratings of managers predict their team performance; the staff is more satisfied with their manager and their job when perceptions of the manager match the manager’s self-perceptions, and more successful managers are less likely to inflate their self-ratings of leadership and performance [8]. Furthermore, leaders who receive 360-degree feedback have shown significant improvement in their follow-up 360-degree surveys several months later on behavioral areas that showed a need for improvement [8]. Additionally, leadership effectiveness may improve by as much as 60% in development programs that utilize 360-degree feedback and coaching [10].

Another positive impact of 360-degree feedback is the encouragement of developmental goal-setting and impression management which are essential aspects of leadership development [9]. However, it should be noted that this improvement is heavily dependent on the presence of post-feedback support through leadership development activities (e.g., educational programs, coaching, etc.). Taken together, these findings support the value of using 360-degree feedback to assess and improve leadership skills.

Utilizing 360-degree feedback in a professional medical association

Given the limitations physicians face in pursuing continuing education in leadership, professionalism, and interpersonal/communication skills, the present study sought to evaluate the utility of 360-degree feedback as a measure of leadership potential. Self-other agreement, which reflects how one sees him/herself compared to how others see them, on 360-degree ratings may be used as a proxy for emotional intelligence [11] and should therefore be predictive of leadership performance. Prior research has shown that there are four general types of self-other agreement: (1) Over-Estimators, (2) Accurate Estimators-Unfavorable (unfavorable scores on both self-rating and other-ratings), (3) Under-Estimators, and (4) Accurate Estimators-Favorable (favorable scores on both self-rating and other-ratings) [11, 12]. Over-Estimators are were defined as individuals whose self-rating was more than half a standard deviation (SD) from the mean rating of others in a self-favoring direction; Under-Estimators were individuals whose self-rating was more than half an SD from the mean rating of others in a self-deprecating direction; Accurate Estimators (Favorable or Unfavorable) have self-ratings within + or − half a SD from the mean rating of others. Research findings have generally shown that the first two types of estimators perform poorly/are rated poorly in a leadership context while Under-Estimators and Accurate Estimators-Favorable are rated more favorably on their leadership performance. Hence, we sought to evaluate this paradigm with physicians in leadership roles in a professional society. The central thesis for the project was that Under-Estimators and Accurate Estimators-Favorable will be more favorably rated, and hence show a greater potential for future leadership success.

Method

Participants

All 86 physician members of the Society of American Gastrointestinal and Endoscopic Surgeons (SAGES) who held leadership roles between 2012 and 2013 (board members, committee chairs, etc.) or recently had been nominated for future leadership roles were invited to participate in a 360-degree survey feedback-based leadership development program. This invitation was accepted by 63 (73%) of the potential participants. This number represents the final sample size for the project.

Measures

The 360-degree survey used was a slightly modified version of the Leadership PULSE 360® Survey to make all items applicable to interactions that would occur in the context of SAGES business. The PULSE 360® Survey is a widely used 360-degree feedback instrument that provides physician-leaders and physicians with a behaviorally anchored assessment of both interpersonally motivating and disruptive behaviors as well as impact on others in the workplace, when compared to normative reference group [13, 14]. Among healthcare professionals the PULSE 360® Survey shows strong internal consistency reliability with alphas >0.70 for all scales. Additionally, inter-rater agreement is also high with intra-class correlations above 0.50 across rater groups [13, 14].

Procedure

Each potential participant was emailed a cover letter from the then-current and past President of SAGES letting them know that the society was participating in a research project on leadership and that—if he/she was willing to participate—there was a hyperlink provided at the bottom of the letter. The hyperlink took the participant to the informed consent/formal agreement to participate webpage. This project was conducted under the auspices of the Cambridge Health Alliance IRB, and the informed consent made participants aware of the general research goals, plan, risks, and potential benefits. Upon electronically signing the informed consent, each participant was automatically prompted to complete the following steps in order to participate in the research project:

  1. 1.

    The participant was instructed to select 10–20 SAGES colleagues (fellow physicians and/or SAGES staff) with whom he/she had worked with the most in the past year of society responsibilities and would like to invite to provide him/her with feedback about his/her leadership style.

  2. 2.

    The participant was informed that his/her list of selected raters would be sent to the immediate past President of SAGES and Executive Director, who were designated to act as the “validators” in order to reduce any participant selection bias. The validators had the option to add raters (up to 100% of the number chosen by the participant), but the software prohibited them from dropping any raters.

  3. 3.

    Once the rater list was validated, the Leadership PULSE 360 survey was automatically emailed to each rater on the list along with a request to the participant to complete a self-survey. Each rater was given 14 days to complete the survey and was automatically sent periodic reminders. Raters were informed that they had the option to decline completion of the survey should they feel uncomfortable or felt they had not interacted with the participant enough during the past year to complete the survey. The average number of raters selected for each participant was n = 19.8, and the average response rate was 58%.

After the survey cycle was completed for each participant, the feedback data were analyzed, and free-response comments were automatically edited to protect the anonymity of the raters. Feedback reports were reviewed by the immediate past president and the Executive Director for approval and were then made available to the individual participant. Any participant who requested feedback interpretation or coaching was offered this service free of charge.

Results

Demographics

The final research sample included 63 of SAGES leaders/soon-to-be leaders, which was 84% male, average age was 49 years old, average membership tenure of 15 years, average number of committees-years served of 32 (reflecting membership on several committees simultaneously over a number of years), and average number of committee chair appointments of three. A total of 87% of the participants were from academic medical centers or community hospitals in the United States, and 13% were from Canada. Finally, 72% of the participants currently held leadership positions at their home medical institution.

Self-other agreement estimation

Self-other agreement was estimated using the scheme outlined by Atwater and Yammarino [12]. Since the Leadership PULSE 360 contains both positively and negatively scored items, all negative items were reverse-scored before determining the self-rating minus mean others’ rating difference for each item. The computed difference was then averaged and compared to the standard deviation of mean others’ rating to determine for each participant whether he/she fell into the Over-Estimator, Under-Estimator, or Accurate Estimator-Favorable category. Based on this approach, we were able to categorize participants as n = 3 Over-Estimators, n = 16 Accurate Estimator-Favorable, n = 18 Under-Estimators, n = 26 No Self-Rating Completed (did not complete a survey self-rating), and there were no Accurate Estimator-Unfavorable categorizations identified.

Analyses of variance

A one-way analysis of variance comparing the four self-other agreement categories was conducted to evaluate for significant mean differences in others’ ratings across the PULSE 360 survey questions and composite scores. Based on this ANOVA several significant F values were discovered leading to Tukey’s LSD post hoc comparisons (See Table 1).

Table 1 Self-other agreement category group comparisons
Full size table

A consistent pattern of results was discovered as Under-Estimators were rated significantly higher than Over-Estimators on 76% (19/25) of the positive leadership behaviors on the survey. Similarly, Under-Estimators were rated significantly lower on 35% (11/31) of the negative leadership behaviors. Also, Under-Estimators were scored significantly higher on the Leadership-Teamwork Index composite PULSE 360 score (overall rating based on PULSE item scores) than Over-Estimators. Table 1 provides the results of the ANOVA, and Table 2 presents the post hoc comparisons.

Table 2 Post hoc comparison of self-other agreement category group means
Full size table

A similar pattern of results occurred when comparing Accurate Estimators-Favorable with Over-Estimators. Accurate Estimators-Favorable were rated significantly higher than Over-Estimators on 48% (12/25) of the positive leadership behaviors and also on the Leadership-Teamwork Index composite PULSE 360 score. Accurate Estimators-Favorable were also rated significantly lower on 29% (9/31) of the negative leadership behaviors. There were no significant differences found in the ratings received by Accurate Estimators-Favorable and Under-Estimators on any of the PULSE 360 behavioral items or composite scores (see Table 2).

Discussion

The findings of the current study provide support for existing research in the area of self-other agreement as well as identify some behavioral relationships that may be unique to the PMA environment and evaluation of leadership. The related literature has consistently demonstrated the importance and value of emotional intelligence to the effectiveness and success of physician-leaders [4, 15–17]. The current study sought to demonstrate that self-awareness of one’s behavior is a key component in helping to understand why physician-leaders’ level of emotional intelligence is pivotal to their future success. We approached this assertion by assessing self-other agreement on a 360-degree survey of leadership and teamwork behavior as a proxy for a more traditional measure of emotional intelligence. The extant research on self-other agreement lead us to speculate that Accurate Estimators-Favorable would be rated most highly on leadership related behaviors by others. This assertion was supported by the current findings as there was a consistent pattern whereby Accurate Estimators-Favorable were scored significantly more favorably than Over-Estimators by others on numerous behaviors. Furthermore, this pattern extended to negative leadership behaviors as well with Accurate Estimators-Favorable being perceived by raters as engaging in more unfavorable discouraging/demotivating behaviors.

Similarly, Under-Estimators (those with lower self-ratings who are presumably more humble) were also rated significantly more favorably than Over-Estimators on a variety of behaviors, but surprisingly did not significantly differ from Accurate Estimators-Favorable in any of the ratings they received. While the current research provided support for our central hypotheses, we had posited that there would be a hierarchy of leadership/teamwork effectiveness ratings with Accurate Estimators-Favorable at the top followed by Under-Estimators, and then Over-Estimators. The unexpected null finding regarding differences between Accurate Estimators-Favorable and Under-Estimators could be interpreted in several ways: (1) Within the context of a PMA and its associated leadership structure, humility may be a more impactful leadership quality than in other settings (i.e., traditional business, hospital setting, etc.); (2) Given the mostly positive ratings received by the SAGES leadership, the null finding may be artificially driven by lack of variance in leadership rating (i.e., the traditional criteria for identifying an Under-Estimator [more than 0.5 SD below Others’ ratings] [1] may not be appropriate which could potentially lead to a larger percentage of the sample classified as Accurate Estimators-Favorable); and (3) Leadership effectiveness among PMA physician-leaders may be mitigated by overconfidence in one’s abilities, but be similarly bolstered by self-aware/self-deprecating leaders because that behavioral difference has only internal consequences for the leader.

Despite the supportive preliminary findings already discussed, there were limitations to the current research that may need be addressed in future studies to more comprehensively tackle the question of how 360-degree surveys can be leveraged toward the prediction of leadership potential and effectiveness in physician PMAs. One limitation which has been previously touched on was the lower variability in the PULSE 360 ratings received by our sample. For the most part, the SAGES participants were rated very favorably by their colleagues which lead to only n = 3 participants classified as Over-Estimators, which is generally consistent with other physician-leadership groups participating in 360-degree feedback. However, despite this limitation we were still able to classify n = 16 Accurate Estimators-Favorable, n = 18 Under-Estimators, with n = 26 unclassified due to non-completion of the self-rating and discover significant differences in their received rating patterns. In an ideal scenario, we would have had a roughly equivalent split of about n = 21 Over-Estimators, Under-Estimators, and Accurate Estimators-Favorable. It is purely speculative, but perhaps a majority of the non-classified participants would have turned out to be Over-Estimators given the similar less favorable scoring profile we observed for this group. Future research will need to address this limitation by more rigorously controlling for the representation of each type of leader in follow-up studies.

Another limitation of the current project is that the sample was limited to only the members of the SAGES physician PMA. Our ability to extend these findings beyond the particular culture and context of the SAGES PMA is diminished, but there is no reason to believe that SAGES is particularly different in their organization, structure, or culture than other physician PMAs. Therefore, the findings still provide some interesting preliminary inferences about how leadership effectiveness and potential is gauged within a PMA and how existing professional development tools like 360-degree surveys might be used to help PMA boards better identify and assess those who may be more successful in leadership roles. Future research should include a variety of physician PMAs to determine if our preliminary findings are generalizable in order to formalize trends and patterns in self versus other assessment of leadership behaviors that can be used to help leadership development within the PMA.

Stepping away from the quantitative findings of the current study, we were in a unique position to also evaluate more casually the opinions of the participants concerning the feedback they received as a result of participating in the study, as well as how some board members of the PMA felt about the findings because one of our investigators is a member of the senior leadership structure. These conversations revealed both the pros and the cons of the methodology. On the one hand, the feedback was not surprising, whether positive or negative, which is a similar response observed in physician-leaders when reviewing the feedback reports of their department’s physicians in the clinical setting. Yet the ability to objectively quantify these suspected “issues” was both helpful and reassuring to some since it is easier to change what is known. At the same time, identifying differential behavior toward society staff versus colleagues was considered valuable information, since the progress toward completing the goals of a PMA really depends on a strong working relationship between physicians and PMA staff. The survey also revealed strong leadership skills in a few physicians not well known to the senior leadership of the society. This represents opportunity for advancement and may in fact be the strongest case for conducting these reviews in the first place. There were also some concerns. Although the vast majority of the feedback was positive, the personal interactions in these groups can be episodic and the depth of experience for certain scores given could be weak, allowing the possibility that a recipient of negative feedback could rationalize why results might not be applicable. In addition, although there is no formal employment relationship between the society and its members, a few individuals who received negative feedback expressed some concern whether it could impact future employment opportunities. Finally, some of the physicians who received negative scores needed some clarification about the difference between criticism (opinions they take personally) and feedback which provides the opportunity to reflect upon how others view them and make positive changes to become a more effective leader.

Medical societies that consider initiating 360-degree leadership reviews on their all-volunteer leaders may be rewarded by even higher levels of performance by those who use the feedback to optimize their performance in this highly specialized setting, perhaps avoiding some of the disappointing events that have beset a few of the large PMA senior leaders in the past. Finally, as in the clinical setting, many physicians hoping to rise through the ranks of their PMA operate in an environment devoid of meaningful feedback, yet commit what may be hundreds of hours a year volunteering to improve the care of patients and the work life of their colleagues. The benefits of formal feedback are the very least they deserve for their efforts.