As LLMs mature, we need structured approaches to develop with them as a team. In this post, we’ll explore a framework for test-driven development of LLM-powered applications. TDD for LLMs can lead not only to faster development time and fewer errors, but also create a virtuous cycle of continuous improvement of the model as more undesirable behaviors are detected and folded into the tests

Part of an ongoing series highlighting insights from papers that have contributed to the development of best practices for production ML

Part of an ongoing series highlighting insights from papers that have contributed to the development of best practices for production ML

Part of an ongoing series highlighting insights from papers that have contributed to the development of best practices for production ML

Part of an ongoing series highlighting insights from papers that have contributed to the development of best practices for production ML

Part of an ongoing series highlighting insights from papers that have contributed to the development of best practices for production ML

Part of an ongoing series highlighting insights from papers that have contributed to the development of best practices for production ML

Part of an ongoing series highlighting insights from papers that have contributed to the development of best practices for production ML

Part of an ongoing series highlighting insights from papers that have contributed to the development of best practices for production ML

Part of an ongoing series highlighting insights from papers that have contributed to the development of best practices for production ML

Part of an ongoing series highlighting insights from papers that have contributed to the development of best practices for production ML

Part of an ongoing series highlighting insights from papers that have contributed to the development of best practices for production ML

Part of an ongoing series highlighting insights from papers that have contributed to the development of best practices for production ML

When you and I look at a software application, we see more-or-less the same thing. Maybe the order of the items presented to you in a recommendation carousel is different, but the structure of the page is the same.

Many machine learning practitioners advocate for **the importance of ethical AI.** But in practice, few ML teams put even basic fairness / bias checks and balances in place.

Normally, we think of the ML process as model-centric: we iterate on the model until it performs well on a given dataset. Data-centric AI inverts the model-centric process by bringing data into the iteration loop. We improve the quality of the dataset, which in turn translates to a better model.

Placing too much importance on "data drift" is one example of the bad advice you'll often hear about model monitoring on the internet. Drift can hurt your models, but it's not guaranteed to. There's no way to know whether a KL=0.17 will have a big impact on performance.

Machine learning is undergoing a modularity revolution. ML purists are trained to appreciate end-to-end models, like a self-driving car that maps raw sensor inputs directly to motor commands and is trained directly to get from point A to point B, avoid collisions, etc. However, in the real world, increasingly ML systems are composed of several (or even thousands) of models working together.

One frustrating truth about production machine learning is that models, especially large ones, tend to fail in spectacular and unexpected ways. Suppose your model makes a mistake, like answering "who is the president of the US" incorrectly, or suggesting something offensive.

Your favorite AI breakthrough of the last 1-2 years was probably trained on a massive amount of web-scraped data. Before a few years ago, you needed to employ an army of annotators would have to get labels for all it. Now, researchers use techniques like self-supervised learning that give us "labels" for free as we scrape.

Training an ML model is easier than ever, but building an ML-powered product isn’t. ML-powered products need their own dedicated tooling stack.

You shipped your machine learning model, and it’s starting to interact with real users. Congratulations on not being part of a (possibly made up) statistic about 87% of models never making it into production.

One of the biggest misconceptions I hear from laypeople about AI is the belief that machine learning models get smarter as they interact with the world.