|
To be clear, the value of services like Legora and Harvey was not as a replacement for legal-specific services like Westlaw. LLMs are absolutely abysmal at legal research and analysis, and every week we are seeing reports of some lawyer being called out publicly for submitting a brief to a court with hallucinated case citations. This has happened to lawyers at all levels, including lawyers at big law firms. If you ask an LLM questions about very basic principles of law, it will do a decent job, but anything requiring nuance or specialization, they will spit out things that sound plausible, but are not correct. And this is not likely to change in the future, as the legal market is so small and niche that the leading makers of LLMs have put legal analysis near the bottom of their list of priorities in terms of improving model performance. There is very little if any effort by the major LLM companies to curate sources of additional high-quality legal training data or fine-tune their models to improve performance on legal tasks. Law is also a field with very low tolerance for error, where tiny mistakes can have big consequences, and getting the models to perform well under these constraints would require a lot of investment without a sufficient payoff. The true reason big firms are buying Harvey and Legora subscriptions is simply to use an LLM for LLM-type tasks, like document review, spotting issues in user-provided documents, and other things that LLMs do well. True, services like Harvey and Legora have lots of cool templates and features for legal work, but you will find that most of the people who use these services in these firms use them much the same way they'd use ChatGPT, Claude, or any other AI chatbot. The reason law firms can't just use ChatGPT or Claude is that they can't allow confidential or privileged client data (such as documents provided through the prompts) to be stored and hosted on a third party service like ChatGPT or Claude, as these companies may have to turn over client data in response to subpoenas, and depending on the type of LLM account you have, these companies could use your user prompts to train future models thus risking leakage of client data to third parties and potential privilege waiver. Services like Harvey and Legora solve this problem by accessing the LLMs through APIs, and all client data, prompts/responses, etc., are stored encrypted on Harvey or Legora servers and protected by keys held by the customer. For many law firms, this is 95% of what they're paying for. The big challenge "Mike" presents to services like Harvey and Legora is that it exposes how little additional value they offer over ChatGPT or Claude, for the vast majority of law firms. A system like "Mike" can provide the same security benefit at basically $0 cost, and can be hosted on the law firm's own internal servers. This is going to put a lot of pricing pressure on services like Harvey and Legora; law firms are notoriously cheap when it comes to IT and software spend and will switch quickly if cheaper alternatives arise. This confirms that Harvey and Legora are going to have to sell their services based on the value they add to lawyer productivity, and not just on being a protected wrapper around GPT or Claude. |