Keywords

Legal departments around the world are busy creating, implementing and refining performance evaluation models to better assess how their legal departments are satisfying client demand and advancing the business objectives of their companies. At the root of this effort is the basic concept of measurement. Companies are utilizing quantifiable performance ratings to evaluate the effectiveness of the legal function. From meeting budgets, managing work-flow, and providing responsive services to improving cross-functional communication and harnessing efficiencies from outside counsel, legal departments all over the world are turning to metrics, metrics, and more metrics.

This development raises the question as to whether general counsels and hiring managers should change the way they evaluate candidates for open positions in their legal departments. Is the traditional method of relying on a standard resume, the in-person interview, and pro-forma post-interview reference checks an adequate way to evaluate whether or not a candidate possesses those attributes that will succeed in a metrics-driven environment? Put another way, is the traditional candidate vetting process in alignment with the methods now being used to measure the overall effectiveness of the legal department? After all, no legal department can succeed in a metrics-driven world if the lawyers who comprise it are themselves improperly vetted on the very criteria the department itself will be measured against.

1 The Traditional Vetting Process

For decades most in-house legal departments have relied on a three-pronged vetting system: (1) evaluation of the candidate’s credentials and experience as revealed on the resume to assess advanced reasoning aptitude and substantive skill set compatibility; (2) the in-person interview to assess cultural fit; and (3) the pro-forma post-interview reference check to make sure there is nothing glaringly wrong with the candidate that the first two parts of the system failed to reveal.

Within these three prongs, most hiring managers placed significant emphasis on only a couple of key components. As for the resume, the single most important factor has always been academic credentials. Sometimes to an exclusionary degree, but always to a significant degree, employers relied on the prestige of the candidate’s law school and his performance within the law school to assess general levels of intelligence and advanced legal reasoning. The more prestigious the better, and the higher the class rank the better. Such credentials served as certification of the candidates’ ability to navigate complex legal issues and also to excel at legal writing and drafting. In short, the credentials established the candidates’ high quality lawyering skills.

The credentials could be augmented by a work experience (including the reputation and prestige of previous employers) that confirmed the above assessment, or the work experience could dilute the value of the credential in certain rare cases. More often the traditional system looked at work experience in a purely utilitarian light. Is the experience relevant to the type of work that the candidate would perform at the company? If it is and the credentials are strong, then the evaluation of the resume often would come to a quick end.

Once the candidate cleared the credential and work experience hurdles, the in-person interview was primarily used for two purposes. First, to gently probe the quality of the candidate’s lawyering skills to confirm the positive assessment. Second, and more importantly, to determine if the candidate’s personality would fit with the company’s culture. Under the traditional method, this was a necessarily vague and subjective assessment which all too often boiled down to whether or not the general counsel or hiring manager felt “comfortable” with the candidate. Whenever such an assessment is made without reference to some defined criteria, many people do the natural thing: they find themselves more “comfortable” with a candidate who is most like themselves. Hiring in one’s own image is among the most common results of the traditional vetting process.

Finally, the traditional method would use the post-interview reference check in a fairly perfunctory way to basically (and briefly) allow the reference to confirm the Hiring Manager’s positive view of the candidate. Many companies traditionally do not even initiate the reference checking process until they have made an offer, making the offer conditional on the positive reference. Given that it is the candidates themselves who provide the references, a negative reference is extremely unlikely. In fact, in the over 25 years of my experience in the legal search business and the thousands of placements made during that period, I can only recall one instance where an offer was rescinded as a result of a negative reference. In short, as traditionally used, the post-interview reference check is a pro-forma waste of time.

Given the limitations of the traditional method, it is hardly surprising that some companies are moving to a different system utilizing different tools to assess potential candidates. Also not surprisingly, many of these tools are beginning to incorporate the very metrics that now help to define a legal department’s success. In short, companies are now beginning to measure the suitability of potential candidates.

2 What to Measure

The Association of Corporate Counsel (“ACC”) provides on their website a sample Legal Department Client Satisfaction Survey. While every company would want to customize their own survey, the ACC survey does an excellent job of setting forth those categories of legal department performance that are of critical importance to the client. The survey then asks the client to rate the legal department on a numeric scale in each category. Our own experience as legal search consultants confirms that the categories identified by ACC are the ones emphasized over and over again by our corporate clients. What are the categories that legal departments will be judged on?

  1. 1.

    Aptitude and Drafting Skills. This is the bread and butter of lawyering. Clients look to their lawyers to have the ability to provide, in essence, the correct legal answer to a wide variety of problems that arise in the business. Lawyers must be capable of advanced legal reasoning, they must know the law, and, because so many in-house positions are transactional in nature, clients look to lawyers to effectively memorialize business agreements. By “effectively” the clients mean that the lawyers will draft agreements that accurately reflect the business understanding while, at the same time, shielding the company from undue risk, ambiguity, and misunderstanding. The ability of lawyers to deliver the core competencies of lawyering is a primary criterion that clients will assess, and one the lawyers themselves often believe that clients are not necessarily qualified to measure.

  2. 2.

    Risk Assessment and Business Literacy. One recurring concern that clients often express is that their in-house lawyers are not “business savvy” enough and do not place their legal analysis within a larger business context, which would help the client to appropriately assess various risk factors. The ACC Client Survey poses this question in multiple ways, including asking the client to rate the lawyers on the following proposition: “Demonstrating commitment to helping me find a way to achieve my business objectives instead of just saying -no-.” Of course, it is not possible for lawyers to score well in this category unless they have a deep understanding of the business itself. In turn, it is not possible for lawyers to possess such an understanding unless they are familiar with basic business and accounting concepts. Clients will measure a lawyer’s aptitude in this area.

  3. 3.

    Communication Skills. Every in-house client survey I have ever reviewed contains multiple questions concerning a lawyer’s ability to effectively communicate within the company. This means being able to explain complex legal concepts in terms that non-lawyers can understand. It also means being honest, straight-forward and candid with clients. Lawyers will be judged on their ability to effectively communicate in three basic settings: the group meeting, one-on-one meetings, and brief written summaries. High competency in aptitude and risk assessment/legal writing skills will be wasted if the lawyer is unable to communicate to the client in ways the client can understand.

  4. 4.

    Responsiveness/Time Management. Time and again, clients will emphasize the importance of a lawyer’s responsiveness, which is another way of saying that they value lawyers who can effectively manage their time and prioritize their work in a way so that the client receives the advice and counsel they need in a timely manner. The threshold requirement is one of accessibility. Lawyers cannot be responsive to clients if they are not accessible. But it also means that lawyers must effectively manage clients’ expectations. Over-promising and under-delivering in terms of responsiveness will result in a low rating.

  5. 5.

    Cost Containment and Managing Outside Counsel. Legal services are expensive, and lawyers are increasingly held accountable for containing the cost of these services. Of all the categories which are subject to numeric rating, this is perhaps the easiest to measure. There is a budget, and the lawyers will be held accountable for meeting it. The single biggest variable in doing so is the cost of outside counsel. In order to succeed in the in-house setting, lawyers must be able to take advantage of the expertise outside counsel provides while staying within budget. How they do this is not typically visible to the in-house client. But whether or not they do it is painfully obvious. Bottom lines simply don’t lie.

  6. 6.

    Collaboration and Teamwork. This category is perhaps the most subjective of all, and it is the one that typically produces asymmetrical results on client surveys. But it remains one of the most critical. Some constituencies within a company might think a lawyer is collaborative and an effective teammate. Others quite simply may find it hard to get along with a particular lawyer (and vice versa!). Nevertheless, this criterion looms large in the minds of the clients and is a prominent feature of any assessment.

So then, the question arises, how does one measure potential candidates’ ability to achieve a high score on client surveys once they become an integrated member of an in-house legal team?

3 How to Measure

A cursory review of the above criteria demonstrates the traditional three-pronged method of vetting potential candidates is seriously insufficient. Can a resume, an in-person interview, and a perfunctory reference check provide sufficient insight into a candidate’s ability in all of these categories, or even some of them? Let’s review the measuring tools we have at hand (in the order in which they should be used) to improve the vetting process.

3.1 The Resume

Every evaluation of a potential candidate begins with a review of the resume (we will assume for purposes of this chapter that the resume describes a skill set that is relevant to the open position). And, while limited in its utility, the resume remains the single most (although not exclusive) tool for measuring one aspect of aptitude: advanced legal reasoning. The experience of most general counsels and hiring managers is that academic credentials and law school performance are reasonably reliable indicators of high achievement in the area of advanced legal reasoning. A good law school and strong performance within law school generally correlate to the ability to navigate complex legal concepts and wed them to complex fact patterns to arrive at sound legal results. After all, in order to achieve these credentials, candidates have already gone through a battery of advanced reasoning tests to be admitted to law school and to perform well once they are there.

However, the resume is a poor measuring device to assess almost every other attribute that will be important to achieving success in a legal department. Many general counsels are tempted to also assume that, if the candidate’s early career has been spent at a top-flight law firm, s/he will necessarily have acquired the training to become an adept drafter of contracts and other legal documents. Unfortunately, the correlation between high prestige law firm training and high aptitude for drafting skills is substantially more tenuous than the correlation between advance legal reasoning and academic credentials. As a result, many of our clients now look to a customized writing exercise (discussed below) as better tool for measuring drafting performance.

Many clients are also tempted to look at the work history on a resume and draw conclusions about a candidate’s employment stability. Lots of moves tend to be a red flag for many clients, suggesting that the candidate might be a job-hopping malcontent. However, in today’s world, there are so many perfectly legitimate reasons for cycling through several positions that drawing such a conclusion from the resume would not be fair to the candidate.

And, of course, with respect to every other attribute that will be measured (risk assessment/business literacy, communication, collaboration, time management, etc.), the resume is practically useless as a vetting tool. Fortunately, there are other tools available to measure these other important qualities.

3.2 The Written Exercise/Drafting Skills

It is important at the outset to distinguish the written exercise from a writing sample. Many clients (although certainly not the majority) do ask candidates to provide a writing sample, typically after the initial interview. This request gives the candidate wide latitude to choose a writing sample that may or may not have been written under a tight deadline, may or may not have been written with the help of others, and may or may not have been edited by a third party. In short, the writing sample may not either be representative of the type of drafting the candidate will be asked to undertake in the company, or even be substantially the candidate’s own work product. There is a much better way.

We are seeing a clear trend among our clients to ask candidates to take a written exercise (many still shy away from the word “test”). The exercise can take several forms. Some clients ask the candidates to mark up a draft agreement that the client may have purposefully sprinkled with errors, both substantive and typographical. Other clients present the candidate with a term sheet and ask the candidate to draft an agreement reflective of the terms. Each client can determine what is the best type of exercise to present, but it is important that the exercise be designed to reveal drafting skills of the candidates that are relevant to the work they will be asked to perform. Typically, the candidate is given a time deadline in which complete the exercise (often the candidate will be asked to block out a two hour window where the exercise will be emailed to them and they will have to email it back within the two hours).

Our clients who use these written exercises have found them to be hugely valuable vetting tools that are most appropriately administered before the in-person interviews (although potentially after a phone screen). These are so valuable that we also recommend that the clients administer another much briefer written exercise to measure the candidate’s ability to write internal communications to non-lawyers explaining a legal concept. This is usually a brief two or three paragraph email the candidate might be asked to write to the head of sales explaining new revenue recognition rules, for example.

3.3 Psycho-Metric Testing: Advanced Reasoning and Personality

These types of tests have been around for a long time and have made very few in-roads into the vetting process of lawyers. Part of the reason may be that the credentials on the resume are usually viewed as sufficient for establishing advanced reasoning capabilities. Another reason is that lawyers are skeptical of personality tests generally and feel they are too generic and easily gamed.

These objections have some force to them but we believe these tests can be helpful in certain circumstances, particularly in those companies where the tests have been found to be useful in vetting candidates for other functions. The personality tests can help identify potential strengths or weaknesses in the area of collaboration and teamwork. The advanced reasoning tests can also help to clarify the nature and scope of a candidate’s reasoning abilities, particularly if a candidate possesses less than stellar academic credentials but otherwise appears to have solid and relevant experience. Rather than simply rejecting such a candidate, an advanced reasoning test might prove useful in establishing that candidate’s capabilities. Each client will have to determine for itself whether the use of such tests is appropriate, but they should be considered as part of the mix.

3.4 The Interview

Now comes the truly hard part. There is simply no way around the acknowledgement that the interview is the single most important, and single most squandered, tool for vetting candidates. Properly conducted, the interview can shed light on nearly every category of importance. Unfortunately, more often than not, the interview devolves into a rambling exchange of potentially relevant (or not) anecdotes. The interviewer either achieves that level of “comfort” with the candidate or not. More often than not, the main criteria that are so important to the metrics-driven environment in which we operate are barely explored.

Interviewers need to have a plan for each interview. Our own experience teaches us that the interview is particularly valuable as a measuring tool for the following criteria: risk assessment/business literacy; communication skills; responsiveness/time management; cost containment/management of outside counsel; and collaboration and teamwork. Of course, the interview can also probe the issues of advanced reasoning and the candidate’s understanding and mastery of relevant legal principles.

Time and space constraints prohibit setting forth a comprehensive list of all the questions that could be asked in an interview to elicit valuable information, but when formulating an approach to the interview, the interviewer might consider alternating between more general questions and more specific ones. In my own experience, I have found that when testing for risk assessment and business literacy, open-ended questions tend to elicit very useful answers. For example, I might ask the candidates to generally describe the business models of the various companies where they have worked. In their answers I look for an understanding of how the products are developed, how they come to market, how they are priced, how they are sold and distributed. In short I’m seeking from the candidate evidence that the candidate understands the business context within which the legal advice is rendered. Similarly, I might ask an open ended question to a candidate who worked on a M&A deal: “Why did your client buy that company?” A surprising number of candidates can struggle with that answer.

Conversely, I have found more specific questions are useful when measuring the criteria of responsiveness/time management and cost containment/management of outside counsel. For example, I might give the candidate a list of three or four projects that could be on their plate and ask them to specifically prioritize their approach and give me an idea of how long they might take to complete each task. I will also ask them to give me hard numbers on the cost of outside counsel for various assignments and what they think is reasonable for outside counsel to charge. Of course, they must demonstrate an understanding of the market for fixed fee arrangements and discounts.

When it comes to communication, there is nothing like the art of listening. A candidate won’t be good at communicating if they aren’t good at listening (and an interviewer won’t be good at measuring a candidate’s communication skills if they aren’t good at listening either). The threshold issue is this: is the candidate listening to the question and responding to the question that is asked rather than answering some unasked question that the candidate would prefer to answer? If they are on track, then I have found it useful to ask a very direct question: “Pretend I am on the sales team, please describe to me the potential risk to the company if we agree to this indemnity provision (insert an appropriate indemnity clause).” Do they respond clearly, concisely, respectfully and in a manner that a non-lawyer can understand?

It is important to remember that communication within a company takes place in three principal settings: the group meeting; one-on-ones; and written communications. We have already tested for written communication and the interview handles the one-on-one. How about the meeting setting? How can we measure whether a candidate will function effectively in the group meeting with all of its unpredictable conversational flows, protocols, and frustrations! The group interview is the answer. As part of the interview process, many of our clients have found that in addition to one-on-ones, it is valuable to subject the candidate to a group interview to see how they handle over-talk and whether they can be inclusive and respectful when fielding comments or questions they might find irrelevant. Do they interrupt? How do they handle interruption? Can they effectively insert themselves during cross-talk? Will they let people have their say or try to shortcut the conversation? Success in meetings is a key component that informs how the client will evaluate a member of the legal department. The interview process should attempt to partially replicate that setting to see how the candidate manages it.

Collaboration and teamwork are the most difficult criteria to measure. As noted above, psycho-metric personality tests can be potentially useful. The interview can also give you a sense of a candidate’s collaborative skills, but it is very unlikely that a candidate will overtly demonstrate bad teamwork tendencies. Since this is such an important criterion, rather than just trust to hope, it is important to turn to third party testimony. This is the one area where the under-used reference check can be very valuable.

3.5 The Reference Check

As noted above, it is important to conduct the reference check at a point in the process where it can be of use, rather than as an afterthought once the offer has been extended. Ideally, in addition to the references that the candidate provides, it is very useful to “back channel” the candidate with others who may have worked with the candidate but who are not on the official reference list. Of course, this can be tricky, and clients must be mindful of keeping a candidacy confidential. But if recourse to an unofficial reference is available and appropriate, it is wise to contact that reference.

Once the list of references is complete, what is the best way of utilizing this resource? Again, it is important to have a plan and prepared questions rather than the open-ended “how was Jane Doe to work with, was she a good colleague?” We have found that the best approach is to ask the reference to cite specific examples of a candidate’s performance in each of the relevant categories. For example, “can you give me an example of an instance where Jane Doe ran into some uncooperative colleagues on a particular matter and describe how she navigated to a successful outcome?” “How does she perform in a meeting context, and can you give me an example that illustrates her style of communication within the meeting context?”

3.6 Damning with Faint Praise

The way references answer these questions can be very illuminating. Very rarely will they say anything overtly negative. However, if they can’t readily retrieve any examples that illustrate the topic at hand or if they don’t seem very interested in even trying, that can be an indicator of a lukewarm feeling about the candidate. Of course, they just may be too busy to invest the time in giving you a substantive answer in which case it is best to move on to another reference. More often, a reference who is not that enthusiastic about a candidate, will heavily qualify their praise of a candidate. A heavily qualified reference is usually a warning sign.

4 Conclusion

We all know that bad hiring decisions are costly, both in economic terms and in the reputational currency of the legal department. Getting it right is probably the single most important responsibility of a general counsel. Good hiring decisions require two things: (1) a thoughtful process that aligns the criteria for selection with the criteria to which the legal department as a whole is subjected; and (2) a core acknowledgement that tools other than the judgment and instincts of the general counsel and/or the hiring manager are required. Establishing such a process takes a bit of doing and a modest investment of time and effort and training. In larger companies, the human resources departments can help in creating customized screening and assessment tools. Many larger companies are also adding legal operations managers who bring consistency and efficiency to the recruiting process. More and more companies are discovering that once such a recruiting infrastructure is in place, the legal department and the company as a whole will benefit immeasurably.

Liquid Legal Context

By Dr. Dierk Schindler, Dr. Roger Strathausen, Kai Jacob

As we have taken deep dives into creating different, entrepreneurial-minded environments for lawyers (Tumasjan and Welpe), as we have learned about the unstoppable force of change and opportunity that comes with legal tech (e.g. Markfort, Wojcik and Zetterberg, Bues and Matthaei) and as we have learned about the different and new types of roles (Brenton) and skills that are needed, there is a glaring question in front of us: How do we hire against a profile that needs to encompass all of the skills needed to work in this new environment?

Escher taps into the wealth of his experience in advising clients globally in hiring legal talent. He calls an end to the traditional three-pronged approach to hiring: credentials, culture fit, pro-forma reference check. He is brutally honest when he sums up that, too often, we basically look at the lawyering skills plus a gut feel on the culture fit, based on a personal conversation, when we take a hiring decision.

Escher offers a very tangible and practical guide as to “what to measure” in the hiring process. He also adds recommendations on how to do it. Reading his conclusions and recommendations, they are in line with what we can derive from the other articles in terms of skills and mindset required. When technology shaves off the repetitive and standardized part of the jobs in the legal industry, next to technology, talent becomes an even more important asset than ever before. Then, a planful, objective and well developed approach is an obvious key to success.