Introducing the Prompt Engineering Toolkit A well-crafted prompt is essential for obtaining accurate and relevant outputs from LLMs (Large Language Models). Prompt design enables users new to machine learning to control model output with minimal overhead. To facilitate rapid iteration and experimentation of LLMs at Uber, there was a need for centralization to seamlessly construct prompt templates, manage them, and execute them against various underlying LLMs to take advantage of LLM support tasks. To meet these needs, we built a prompt engineering toolkit that offers standard strategies that encourage prompt engineers to develop well-crafted prompt templates. It also provides functionality to enrich these templates with context obtained from RAG (Retrieval-Augmented Generation) and runtime feature datasets. The centralized prompt engineering toolkit enables the creation of effective prompts with system instructions, dynamic contextualization, massive batch offline generation (LLM inference), and evaluation of prompt responses. Furthermore, there’s a need for version control, collaboration, and robust safety measures (hallucination checks, standardized evaluation framework, and a safety policy) to ensure responsible AI usage. This blog post gives an overview of the prompt template lifecycle, the architecture used to build the prompt toolkit, and the production usage of the toolkit at Uber. Goals When we built the prompt toolkit, our goals were that users at Uber could: [object Object], [object Object], [object Object], [object Object], [object Object], [object Object], [object Object] Prompt Engineering Lifecycle The prompt engineering lifecycle consists of two stages: the development stage and the productionization stage. Users interact with the prompt toolkit during each of these stages. Development Stage The development stage has three phases: LLM exploration, prompt template iteration, and evaluation. In the LLM exploration phase, users interact with a model catalog and Uber’s GenAI Playground. A model catalog is a model repository that contains all the available LLM models and its corresponding description, metrics, and usage guide. The GenAI Playground helps users explore the capabilities of different LLMs available at Uber and test their use case applicability. Users explore available LLM models from a model catalog and test LLM responses with prompts in the GenAI playground. In the prompt template iteration phase, users pinpoint their specific business needs, gather sample data, create/analyze/test prompts with LLM models and datasets, assess the responses, and make revisions as needed. Auto-prompting suggests prompt creation so users don’t need to create the prompt template from scratch. With a prompt template catalog, users can explore and reuse the existing prompt templates. In this phase, users perform continual experimentation and iteration. In the evaluation phase, users evaluate the effectiveness of the prompt template by testing it with a more extensive dataset to measure its performance. It could leverage LLM as a judge or leverage a customized code-based LLM evaluator to evaluate the performance of the prompt template. Productionization Stage In the productionization stage, users only productionize the prompt template that passed the evaluation threshold on an evaluation dataset. Users track and monitor usage in a production environment, collecting data on system usage. This data can inform further enhancements to the process. Architecture With the prompt engineering toolkit, various components interact to facilitate LLM model deployment, prompt evaluation, and batch inference responses. The toolkit is composed of a Prompt Template UI/SDK that manages prompt templates and their revisions. The Prompt Template UI/SDK integrates with APIs like GetAPI and Execute API. These APIs interface with the LLM models deployed from the LLM Model Catalog. Models and prompts are stored in ETCD and UCS (object configuration storage), which are then used in an Offline Generation Pipeline and a Prompt Template Evaluation Pipeline. UI/Design Users first interact with the prompt toolkit in Gen AI Playground. Gen AI Playground is designed to interface seamlessly with the Model Catalog, a comprehensive repository of large language models available at Uber. This platform allows users to explore detailed specifications, expected use cases, cost estimations, and performance metrics for each model. Within the Gen AI Playground, users can select any model from the catalog, craft custom prompts, and adjust parameters, such as temperature, to Prompt Template Creation The prompt toolkit prompt builder automatically creates prompts for users and helps them discover advanced prompting techniques tailored to their specific AI use case. The auto-prompt builder, built on top of the Langfx framework (an Uber internal Langfx service built on top of LangChain™), follows these steps: 1. Prompt engineering best practices:
