As law firms gear up for another year of Chambers deadlines, Tom Wicker speaks with Chambers Research Director Kush Cheema to pull back the curtain on how the research process actually works.

Hi Kush! Thank you for agreeing to help law firms gain a better into the Chambers research process and how the team uses the information the provide.

Firstly, though, Chambers and Partners was recently acquired by U.S. investment group Abry Partners. Are you able to share any of the ways in which this acquisition will positively impact the various Chambers guides and products?

I would say that it is a thumping endorsement of Chambers’ transformation and our strategy for the future. Just five years ago, we were still primarily focused on publishing physical books and our sales, marketing and events team numbered in single digits. While we knew there was room to dive even deeper into the market and provide more useful data for in-house counsel, we were more constrained in our ability to take those steps.

Since then we’ve greatly enhanced our internal analytics tools, completely redesigned research strategy and operations, significantly increased our visibility and use among in-house counsel, and provided more and more usable insight for law firms. It’s also led to the us opening up an office in New York — planting the flag for Chambers and thoroughly enmeshing ourselves in the most dynamic legal market on the planet.

Looking to the future, I would expect to see even more insight into law firm performance, capability and potential growth and more research into different market segments (such as our new Spotlight rankings) as well as emerging and disruptive technologies and service providers. I would also expect to see much richer and actionable data for in-house attorneys looking to select or vet their outside legal counsel.

Great, thank you. I suspect the market will be paying close attention to what Chambers does next… For now, on to the questions!

How do researchers access written submissions once they are uploaded?
Chambers has designed a bespoke and secure internal database, through which the research team is able to access the full body of law firm submission materials.

Why is it important that firms stick to the 20-matter limit?
Asking for 20 matters allows firms to showcase their most impressive and sophisticated recent engagements while enabling our team to review them in sufficient depth. We have flirted with the idea of a per-lawyer limit for referees or submissions. However, this could lead to us spending the entire year only researching one law firm! It’s also worth bearing in mind that we keep an eye on other sources, including the news, the industry press and law firm alerts (and beyond) to discover what is happening in various markets and which firms and lawyers are involved.

Can the content of a submission be shared between researchers?
Yes, researchers have access to the entirety of a law firm’s submissions – so, for example, a researcher covering the New York M&A market will also be able to access the firm’s submissions for California and Texas, to look at national capabilities. Elsewhere, a researcher reviewing white-collar crime will also be able to review FCPA, False Claims Act and submissions in other related areas.

How are the contact details of client references stored by Chambers and accessed by individual researchers?
Referees data is uploaded into our secure system by the law firms. At this point, using a nifty bit of internal tech, we’re able to automatically match up referees, including linking to this to previous instances of a referee being submitted. Our system detects when multiple firms or practice groups have referred the same person, which allow us to contact them just once, to collect all of their feedback in one go, rather than emailing them multiple times for multiple firms. This data is accessed by our researchers as we begin our outreach, each time research into a new area starts.

Why do some nationwide and regional practice areas not require references?
Regional practice areas don’t require submissions and referees when they’re already largely covered by the main submission. So, for instance, we wouldn’t require a separate submission for some medical malpractice tables, because we will pull information from the litigation and healthcare research we can already access from submissions provided in these areaa.

For nationwide categories, the areas in which we do not require references are those for which we will already have ample state-based coverage. For example, in Chambers USA, we carry out corporate, real estate, labour and employment, and litigation research in all 50 states (and in Washington, D.C.). We also cover areas such as intellectual property, private equity and healthcare in the vast majority of the major markets. Given that we will usually ask referees for wide-ranging feedback during this state-based research, asking them, later in the research cycle, about the lawyers and practice groups they work with nationally, would be treading the same ground.

Do researchers contact all references provided or do they make judgement calls?
We do contact all referees — apart from those who have unsubscribed from our emails or told us they no longer wish to be contacted by us.

Could you explain the three-month ‘moratorium’ for contacting the same client twice and whether this can still happen if the client gives consent? 
A large proportion of referees are referred either by multiple firms or for multiple practice groups within the same firm. I’ve seen instances where some referees have been provided over 30 times by various law firms. To avoid badgering such referees multiple times, which can quickly lead to ill-will towards both Chambers and the firms which have suggested them, we will typically hold off contacting someone for three months after we’ve interviewed them or emailed them, to provide a ‘cooling-off period’. However, if a firm confirms with us that the referee is happy to be contacted again within this period, we will happily waive this rule and get in contact with them.

Can researchers see who the referring lawyer(s) are, or do they wait to see which lawyers the client chooses to highlight?
Both! When starting off a call, the list of referring practice areas and attorneys is in view in front of the researcher. However, we will typically begin with general questions about which lawyers the referee has worked with recently, and what their experiences were — before filling in any gaps afterwards. This typically allows us to hear about wider members of the practice and associates, not just about the referring lawyer.

Are clients offered a choice (i.e. a phone call or online) in how they provide their feedback?
Yes, we will always offer the choice of a web-based survey, an email questionnaire or a telephone call, depending on the referee’s preference.

Are the questions the same, irrespective of the method of feedback?
Effectively, yes – we’re always looking to glean insights into a law firm or lawyer’s technical excellence, client service skills, the depth of a team and value for money. Phone calls, however, tend to be more free-ranging in comparison to questionnaire responses. We usually allow referees to guide the flow of the conversation during a telephone interview.

How important is the size or profile of the client organisation?
To use a lawyerly term, “it depends”. In our immigration or labour and employment research, conducting compliance reviews or handling mass global mobility programmes will be all the more impressive if they’re carried out for a Walmart (as opposed to a smaller company). However, some of the more impressive work in our consumer finance research this year is likely to relate to the representation of tiny FTX as opposed to the big Wall Street banks. What we look for is based on the market and largely focused on the complexity and sophistication of the matter.

Does the seniority of the individual referee matter?
The best referees are the ones with the time to speak with us. We understand that the CLO of Google (or similar) will have a range of other things worry about. Pinning them down for a 15-30 minute conversation about the firms they use would be difficult. In these cases, we would happily take the word of litigation counsel or legal directors as to the opinion of the company.

We greatly appreciate the time that GCs, CLOs and other C-Suite members give to us, and highly prize their opinions. There is nothing I like more than being able to pick the brain of a Fortune 100 GC about their thoughts on external counsel and how that can change given the volatility of a market. But I would recommend that firms be extremely judicious about when (and for which practice areas) they ask such individuals to serve as referees.

How does the researcher know which firms/lawyers to ask a referee about?
All current and previous referrals are on display for the researcher when using our internal system.

How do referees flag new lawyers for consideration by Chambers?
Our questioning is quite general — we ask referees about any lawyers they have worked with before we focus on specific individuals. In this way, new lawyers and, particularly, up-and-comers tend to show up on our radar quite early.

How is feedback on ‘new’ (i.e. unranked) lawyers stored and – where necessary (i.e. if relevant for more than just the practice area being researched at the time) – shared between researchers?
Each new lawyer is created as a new record in our database. Researchers at Chambers are able to ‘tag’ commentary to the correct place for practice areas outside the one they’re covering. For example, if our securities litigation researcher received commentary about a new lawyer who specialises in FCPA work, they’re able to tag that to our body of research for FCPA and notify the researcher covering that area.

If a referee speaks about multiple practice areas and/or law firms, how does Chambers ensure that feedback is accessible by the right researchers?
In the same way as above, our researchers are able to add to the body of feedback collected by any other researcher for any other practice area – so, for example, a U.S. researcher speaking to someone from LATAM Airlines would be able to collect and apportion feedback relevant to our U.S. aviation research, our UK banking research and our Latin America corporate research.

How is feedback shared between differing Chambers guides?
Research findings are pooled across the research team via our centralised database.

How important is feedback from other lawyers when making ranking decisions?
Once again “it depends”. In areas such as white-collar criminal defence, where different lawyers will often take on different roles defending the corporation and defending individual officers and directors, we highly value feedback from lawyers because they can provide us with first-hand accounts of the technical quality and client-facing skills of the other lawyers they are seeing in these complex mandates.

Conversely, peer feedback is less useful in our research into areas such as environmental or employment compliance work, where the relationship is entirely between a client and a lawyer. In practice areas where the top lawyers do encounter each other, their feedback is used as a ‘sense-check’ to help us verify some of our findings.

What are your top tips to firms for maximising the potential benefits of their references – both before and during the research process?
I would encourage law firms to include referees who can speak to the entire breadth of the team working on their matters, rather than a CEO who may initially engage with the relationship partner, but otherwise only have had a few subsequent check-ins and then signed off on the bill. It’s most useful for us to be able to speak to the in-house lawyers who have had first-hand experience working not only with the lead partners but also the associate handling day-to-day matters and any experts from other practice groups — as well as with the relationship partner.

Do also feel free to refer other sources to us as references. While the majority should be clients, we do value opinions from other lawyers, arbitrators and mediators, and other industry insiders.

During the research process, do let us know if a referee who has previously indicated they no longer wish to receive Chambers’ outreach changes their mind and would be happy to be contacted. And do also let us know if referees who expect to hear from us have not heard anything yet.

Finally, please bear in mind that, as some referees will likely be referred by another firm, they may already have been contacted by another Chambers researcher to the one assigned to the particular category or practice area you’re expecting. for this reason, it’s always best to ensure that your referees are looking for an ‘@chambers.com’ address, rather than a specific researcher.