OpenFn AI Assistant: Now with Direct Access to Documentation

Our AI Assistant, which helps you write your job code in OpenFn, now has direct access to OpenFn documentation. This means that the Assistant’s answers will be more accurate and precise, as they’re based on the latest, complete version of the documentation.

You won’t notice this change in the assistant UI, but this blog post will give an insight into how it works in the background.

Introduction to RAG

You can pack a lot of instructions into an LLM prompt. However, you’ll quickly hit the token limit if you’re working with a broader corpus of context, which is exactly the case with software documentation and technical reference materials. Moreover, as you approach the limit, the model will only be able to follow the prompt approximately, potentially mixing up information and forgetting instructions, which might be unacceptable in scenarios that require highly reliable model outputs. Costs for each query will also increase, often prohibitively, especially for long conversations.

Retrieval Augmented Generation (RAG) is one possible solution to this problem. It works on the basis that while there might be a broad corpus of information overall, only specific parts are needed to produce individual LLM responses. RAG first retrieves the relevant pieces of information based on the user input, and then adds that into the LLM prompt as additional context before generating an answer.

First step: Creating a Vector Database

The Retrieval part of RAG is often done through vector search – a standard retrieval method that encodes texts into vectors (also called embeddings) that can be used in a similarity calculation. To use this method, we implemented a database (through Pinecone) that stores OpenFn documentation as short excerpts, each one represented as a vector (produced with OpenAI’s embedding model). This allows us to search the latest database by encoding an input query as a vector with the same model, and select passages from the documentation that are most similar to the input query.

Deciding When and What to Search For

The input query could just be a user’s conversation turn. Vector search is flexible, and we could encode an entire rambling question to search for matches. However, to improve the relevance of the search results, we added an additional step to reformulate the user’s conversation turn into an input query using an LLM, including an optional filter to limit results to a section of the documentation. We also added a step to decide on whether to trigger a search at all, using a cheap short prompt.

For example, the user message “Ok that looks good. After the data transformation with the OpenFn adaptor, how do I then add a step to write the data to a Google Sheet?” could trigger the following processes:

1. Search decision LLM, which will output: "True" (i.e. yes, we should search the docs to answer this).
2. Query reformulation LLM, which will output: {"query": "write data to a Google Sheet", "filter": "adaptor_docs"} (i.e. we should search the adaptor documentation for the best matches for this concise query)
3. Database search, which will return the highest-scoring documentation excerpts for the query.

The selected extracts are then plugged into the main assistant LLM prompt, so that the model can answer the user while referencing the relevant information.

Future Work

Currently, we’re trialling just adding RAG at the beginning of a conversation, but we’re aiming to add it flexibly at any point in a conversation.

We’ll also aim to improve the relevance of retrieved texts, and make these available to users. As the basic assumption of vector search here is that text similarity can be used to approximate the relationship between a question and an answer, there are cases where this fails, and so we’ll be exploring other search methods that make better use of the reasoning power of LLMs. If you’re curious to read more, you can peek at our work on GitHub.

We’re excited about how these improvements will make the AI Assistant even better at helping you write job code in OpenFn. If you have any feedback on this feature or the AI Assistant itself, we’d love to hear what you think! Contact us on community.openfn.org or via email to admin@openfn.org