Why code-testing startup Nova AI uses open source LLMs more than OpenAI

It is a universal truth of human nature that the developers who build the code should not be the ones to test it. First of all, most of them pretty much detest that task. Second, like any good auditing protocol, those who do the work should not be the ones who verify it.

Not surprisingly, then, code testing in all its forms — usability, language- or task-specific tests, end-to-end testing — has been a focus of a growing cadre of generative AI startups. Every week, TechCrunch covers another one like Antithesis (raised $47 million), CodiumAI (raised $11 million) and QA Wolf (raised $20 million). And new ones are emerging all the time, like new Y Combinator graduate Momentic.

Another is year-old startup Nova AI, an Unusual Academy accelerator grad that’s raised a $1 million pre-seed round. It is attempting to best its competitors with its end-to-end testing tools by breaking many of the Silicon Valley rules of how startups should operate, founder/CEO Zach Smith tells TechCrunch.

Whereas the standard Y Combinator approach is to start small, Nova AI is aiming at mid-size to large enterprises with complex code-bases and a burning need now. Smith declined to name any customers using or testing its product except to describe them as mostly late-stage (Series C or beyond) venture-backed startups in e-commerce, fintech or consumer products, and “heavy user experiences. Downtime for these features is costly.”

Nova AI’s tech sifts through its customers’ code to build tests automatically using GenAI. It is particularly geared toward continuous integration and continuous delivery/deployment (CI/CD) environments where engineers are constantly shipping bits and pieces into their production code.

The idea for Nova AI came from the experiences Smith and his co-founder Jeffrey Shih had when they were engineers working for big tech companies. Smith is a former Googler who worked on cloud-related teams that helped customers use a lot of automation technology. Shih previously worked at Meta (also at Unity and Microsoft before that) with a rare AI specialty involving synthetic data. They’ve since added a third co-founder, AI data scientist Henry Li.

Another rule Nova AI is not following: While boatloads of AI startups are building on top of OpenAI’s industry-leading GPT, Nova AI is using OpenAI’s Chat GPT-4 as little as possible. No customer data is being fed to OpenAI.

While OpenAI promises that the data of those on a paid business plan is not being used to train its models, enterprises still do not trust OpenAI, Smith tells us. “When we’re talking to large enterprises, they’re like, ‘We don’t want our data going into OpenAI,” Smith said.

The engineering teams of large companies are not the only ones that feel this way. OpenAI is fending off a number of lawsuits from those who don’t want it to use their work for model training, or believe their work wound up, unauthorized and unpaid for, in its outputs.

Nova AI is instead heavily relying on open source models like Llama developed by Meta and StarCoder (from the BigCoder community, which was developed by ServiceNow and Hugging Face), as well as building its own models. They aren’t yet using Google’s Gemma with customers, but have tested it and “seen good results,” Smith says.

For instance, he explains that OpenAI offers models for vector embeddings. Vector embeddings translate chunks of text into numbers so the LLM can perform various operations, such as clustering them with other chunks of similar text. Nova AI doesn’t use OpenAI’s embeddings and instead uses open source for this on the customer’s source code. It uses OpenAI tools only to help it generate some code and to do some labeling tasks, and is going through lengths not to send any customer data into OpenAI.

“In this case, instead of using OpenAI’s embedding models, we deploy our own open source embedding models so that when we need to run through every file, we aren’t just sending it to OpenAI,” Smith explained.

While not sending customer data to OpenAI appeases nervous enterprises, open source AI models are also cheaper and more than sufficient for doing targeted specific tasks, Smith has found. In this case, they work well for writing tests.

“The open LLM industry is really proving that they can beat GPT 4 and these big domain providers, when you go really narrow,” he said. “We don’t have to provide some massive model that can tell you what your grandma wants for her birthday. Right? We need to write a test. And that’s it. So our models are fine-tuned specifically for that.”

Open source models are also progressing quickly. For instance, Meta recently introduced a new version of Llama that’s earning accolades in technology circles and that may convince more AI startups to look at OpenAI alternatives.