The Legal IT Innovators Group (LITIG) led by CMS’ chief innovation and knowledge officer John Craske has launched a benchmarking initiative that aims to agree an industry-wide methodology for measuring the output of generative AI technologies.
The Legal Industry AI Benchmarking Collaboration will gather views from suppliers, law firms and academia around how to ensure a sufficient level of transparency and standardisation among suppliers and tools.
Speaking to Legal IT Insider, Craske said: “There needs to be more transparency. I see it in same way as when you buy a new car, you get a miles- per-gallon figure based on a defined set of real world circumstances. It’s a defined thing; you have to drive this many miles at this speed and in order to test it, they don’t just do it once but many times. With generative AI there is no transparency, no rule or mechanics, people are just making it up.”
Craske added: “I’m not saying that just because generative AI isn’t perfect, it’s not useful. But if there’s no transparency then it’s just snake oil. We need to be able to compare suppliers and be able to see and measure developments and improvements over time.”
Advertisement
LITIG is the UK’s leading legal IT industry body and its members represent around 90,000 users and £20bn of turnover.
Asked how LITIG will respond to any potential suggestions from suppliers that they can’t be expected to give away competitive secrets, Craske said: “We’re not asking how you got to where you got. It’s about testing the products available to customers and testing the output and how that compares with what is promised.”
Craske is encouraging people who are able to contribute meaningfully to the conversation to sign up HERE
Both in-person and virtual meetings will be held to brainstorm a path forward and we look forward to seeing some of you there.
See also:
Leading legal IT industry body launches due diligence questionnaire for suppliers of AI
https://forms.microsoft.com/pages/responsepage.aspx?id=l7Laja8Rdk-3BMGKHStwL7QDn6KrJ59GsUu-EGgbnlBUQVFEUlpMQTExU1BIUjZHNDdDODhBRThBMi4u