Building LLM Applications with Langchain

Hi there! I'm Elliot (Lee Eun-kyu), and I handle engineering across the board at ONDA.

Lately, the race among big tech firms to dominate large language models (LLMs) — GPT-4, Bard, LLaMA — has been the talk of the industry. Right in the thick of this paradigm shift, open-source small language models and their application projects are showing remarkable growth. The consensus that generative AI and LLM capabilities matter is spreading fast.

LLMs are completely revolutionizing how we've processed text data until now, offering fundamentally new approaches to analyzing, generating, and applying data. With LLMs, we can automatically generate or summarize long-form content on any topic, get proactive answers to questions, and smartly convert unstructured data into structured formats — ideas that were either impossible or painfully complex before are now straightforward to implement.

Recent months have seen a stream of fascinating LLM-powered implementations and experiments. Think of research projects that simulate AI characters interacting in virtual spaces, or initiatives like Auto-GPT — autonomous agents that think, plan, and act on their own to execute tasks.

Watching these ideas that we've all imagined at some point actually come to life, you might wonder how these projects manipulate and orchestrate LLMs.

They feel like they're built on incredibly intricate engineering designs and incomprehensible math — but you'd be surprised how simple the core ideas behind most of these projects really are.

Today I'll introduce you to Langchain: a framework designed to compose and modularize complex LLM chains, making it easy to build applications powered by language models.

What is Langchain?

Langchain is an open-source framework that emerged in October 2022. As of June 2023, it's earned nearly 44,000 stars on GitHub, experiencing explosive community growth and building a thriving ecosystem on top of the framework.

The core concept of Langchain, as its name suggests, is chaining together LLM prompt execution with external actions (calculators, Google searches, Slack messages, code execution, etc.).

Think of it as a framework that lets you string together components with inputs and outputs into flows, then modularize and chain those flows into complete applications. This becomes more intuitive when you look at Langflow, a GUI project for designing Langchain workflows.

Langchain has pre-configured modules, and you mix these modules to weave together components, then set up pipelines between them.

There are many types of modules, and they keep growing as the framework ecosystem evolves. Multiple modules come together to form a component, and those components chain together — like stacking LEGO blocks to build a complete application.

Langchain has many module types, but the most essential ones are:

• LLMs
• Prompt Templates
• Agents
• Tools

In this post, I'll explain what each module does and walk through simple examples showing how they work. Keep in mind that not all Langchain modules operate at the same level (some modules require other modules as prerequisites), and the same module can serve different purposes depending on your use case. Think of it like how the same job title can have completely different responsibilities at different companies.

💡Besides the modules mentioned here, there are many others covering various domains — Memory, Vectorstore, Retriever, Text Splitter, Document Loader, etc. Plus more granular or abstract modules like Agent Executor and Toolkit are supported and continuously being added. If you want more advanced framework usage, definitely check those out.

Let's dive into each module. For the examples in this post, I'll use langchain-js, the official JavaScript port of Langchain.

1. LLMs (Large Language Models)

The LLM module is Langchain's engine. It takes the APIs from different language models or model-serving platforms and provides a normalized interface that other Langchain modules can use.

The LLM you connect to the module can be any type. It could be OpenAI's GPT-4, GPT-3.5, Davinci, or Ada models; models hosted through Hugging Face's Inference API; or LLaMA-based models running locally. As the community grows rapidly, most endpoints are supported, and if needed, building your own interface isn't too difficult either.

You also don't have to use just one LLM in an application. When building your app, you could use the Ada model for simple prompts requiring lightweight inference, a self-hosted Vicuna model for more advanced reasoning, and GPT-4 for Agent modules that need complex reasoning and action planning. Structuring your application this way reduces unnecessary resource waste and cuts costs.

Creating an LLM module instance is incredibly simple. Here's an example creating an LLM module instance using the GPT-4 model:

import { ChatOpenAI } from 'langchain/chat_models';

export const gpt4Model = new ChatOpenAI({
  temperature: 0.6,
  modelName: 'gpt-4',
  verbose: true,
  streaming: true,
});

LLM instances created this way get passed to other modules during their creation and handle prompt execution.

2. Prompt Templates

Prompt Templates are template modules that make it easy to insert specified variables into pre-configured prompts.

Say you designed and use a simple language detection prompt like this:

Detect the language of text in <input></input>. Your answer must be in english. don't use language code, return full name of the language in english.
if the language seems like one of programming language (like code blocks) return "english".
if detected language is ambigous, return most popular one.
if you can't detect language, return "english".

use the following format for your answer:
Detected: <detected language>

Example:
<input>
안녕하세요.
</input>

Detected: korean

User Input:

<input>
이 글의 언어를 감지하려고 합니다.
</input>

To turn this prompt into a reusable template where you can dynamically pass input to replace [이 글의 언어를 감지하려고 합니다.], you'd write code like this:

import { BaseOutputParser } from 'langchain/schema';
import { PromptTemplate } from 'langchain';

const LANG_DETECTION = `Detect the language of text in <input></input>. Your answer must be in english. don't use language code, return full name of the language in english.
if the language seems like one of programming language (like code blocks) return "english".
if detected language is ambigous, return most popular one.
if you can't detect language, return "english".

use the following format for your answer:
Detected: <detected language>

Example:
<input>
안녕하세요.
</input>

Detected: korean

User Input:

<input>
{input}
</input>`;

class LanguageDetectionParser extends BaseOutputParser {
  parse = async (output: string): Promise<string> => {
    const result = output.match(/Detected: (.*)/);
    if (result) {
      return result[1];
    }
    return 'english';
  };

  getFormatInstructions(): string {
    return `Your response should be in following format.
Detected: <detected language>`;
  }
}

export const languageDetectionPrompt = new PromptTemplate({
  template: LANG_DETECTION,
  inputVariables: ['input'],
  outputParser: new LanguageDetectionParser(),
});

The LANG_DETECTION string replaces the parts you want to substitute with {variableName}. To use literal curly braces, double them up: {{}}.

LanguageDetectionParser is an OutputParser that parses the prompt execution result to extract only what you need. In this example, it's parsing just the string following Detected:. If the LLM responds with Detected: korean, the parsed output would be korean.

languageDetectionPrompt is the completed PromptTemplate. Along with the components above, it passes input (the variable name used in the template) as inputVariables.

Prompt Templates declared this way are used directly by connecting to LLMs through LLMChain modules, passed to Agents for execution, or used as inputs by various modules that require prompt inputs.

3. Agents

Agents play the most crucial role in Langchain and handle the most complex and sophisticated reasoning tasks.

The Agent concept is rooted in several key generative AI papers, and it continues improving as new approaches emerge. Here are some important references for understanding Agents more completely:

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• Toolformer: Language Models Can Teach Themselves to Use Tools
• ReAct: Synergizing Reasoning and Acting in Language Models

You don't need to read all of these, but they'll greatly help you grasp Agent concepts and how they work.

In brief, an Agent:

1. Reasons (Thought) about the given Tools and current situation to design the next necessary action for completing its Task (Action planning/Reasoning).

2. Once designed, executes the current necessary Action with appropriate Input.

3. After Action execution, analyzes the results (Observation) and repeats steps 1-3 based on that analysis and previous Action results (Chain of thought).

4. Completes and terminates when analyzing results shows the Task is finished or finishable.

This is how Agents proactively leverage available resources to execute tasks.

Given Agents' complex responsibilities, their implementation isn't one-size-fits-all. Even using Langchain, there are multiple ways to structure an Agent, and the module composition and prompt format vary wildly depending on your use case. So in this post, I'll look at Langchain's official documentation's basic Agent example and explain each component used.

💡Even Agents serving the same role can behave differently depending on Agent Executor implementation. The steps described above apply to the most basic Action Agent — other types exist, like Plan-and-Execute Agents that map out all execution steps upfront rather than just reasoning about the current Action, then execute step-by-step.

import { initializeAgentExecutorWithOptions } from "langchain/agents";
import { OpenAI } from "langchain/llms/openai";
import { SerpAPI } from "langchain/tools";
import { Calculator } from "langchain/tools/calculator";

const model = new OpenAI({ temperature: 0 });
const tools = [
  new SerpAPI(process.env.SERPAPI_API_KEY, {
    location: "Austin,Texas,United States",
    hl: "en",
    gl: "us",
  }),
  new Calculator(),
];

const executor = await initializeAgentExecutorWithOptions(tools, model, {
  agentType: "zero-shot-react-description",
  verbose: true,
});

const input = `Who is Olivia Wilde's boyfriend? What is his current age raised to the 0.23 power?`;

const result = await executor.call({ input });

This is an MRKL (Modular Reasoning, Knowledge and Language; pronounced "miracle") Agent example from LangchainJs's official docs.

The example code looks surprisingly simple compared to what we discussed earlier. That's because the agentType defined as zero-shot-react-description acts as a preset configuration containing many elements needed for Agent composition.

I need to find out who Olivia Wilde's boyfriend is and then calculate his age raised to the 0.23 power.
Action: Search
Action Input: "Olivia Wilde boyfriend"
Observation: Olivia Wilde started dating Harry Styles after ending her years-long engagement to Jason Sudeikis — see their relationship timeline.
Thought: I need to find out Harry Styles' age.
Action: Search
Action Input: "Harry Styles age"
Observation: 29 years
Thought: I need to calculate 29 raised to the 0.23 power.
Action: Calculator
Action Input: 29^0.23
Observation: Answer: 2.169459462491557

Thought: I now know the final answer.
Final Answer: Harry Styles, Olivia Wilde's boyfriend, is 29 years old and his age raised to the 0.23 power is 2.169459462491557.

Running this code, you'll observe the Agent receiving the user's initial question [Who is Olivia Wilde's boyfriend? What is his current age raised to the 0.23 power?] and:

1. Using SerpAPI to Google search for Olivia Wilde's boyfriend

2. Having learned Harry Styles is Olivia Wilde's boyfriend, searching for his age

3. Having learned he's 29 years old, using the Calculator Tool to compute 29 to the power of 0.23

4. Having obtained the calculation result, generating the final answer to the user's question

The Agent autonomously determines and sequentially executes this process.

💡If you're curious how this process works, provide a custom CallBackManager to monitor the LLM model's input/output and analyze it. You can also analyze the code-level operation process by reading the Custom Agents section in the official docs. While Langchain provides extensive functionality, its documentation coverage is fairly limited. Directly referencing the code repository makes it easier to use more diverse advanced features.

If you've used Auto GPT, this operation process should feel familiar. Auto GPT is also designed as a massive Agent built on well-orchestrated Agent chains with various available Tools — so if you've wondered how Auto GPT works, this should help answer some of those questions.

4. Tools

Tools are essentially abstracted functions that Agents can use when executing each Action.

Agents receive a list of available Tools as part of their prompt, along with:

• The Tool's name
• The Tool's Description (what role it performs, how to pass Input, etc.)

interface Tool {
  call(arg: string): Promise<string>;

  name: string;

  description: string;
}

The Tool Interface defined in Langchain looks like this. Simple but flexible — as long as it returns a result as a string, you can perform any operation inside call.

Even complex Agents composed of various modules can be provided to other Agents as Tools. This simple yet powerful Interface is one of the factors that makes Langchain's potential virtually limitless.

The example below uses a simple LLMChain — it's a Calculator Tool designed to sanitize input before passing equation values to the Calculator, ensuring the input is in a format Calculator can understand:

import { LLMChain, PromptTemplate } from 'langchain';
import { BaseOutputParser } from 'langchain/schema';
import { Calculator, Tool } from 'langchain/tools';

const CALC_EQUATION_GENERATE = `
Generate a calculation equation from contents of <input> block. generated equation must be valid numeric calculation equation.
You need to convert words to numbers, if feasible.
Do not try to calculate the equation, just generate it.
If the input cannot be converted to a valid equation, return <EOF/>.
If the input is already a valid equation, return itself.
Provide your response in <output> block.

Example
\`\`\`
<input>
(1 billion plus 1 million) * 12
</input>

<output>
(1000000000 + 1000000) * 12
</output>
\`\`\`

Now, Here's your input.

<input>
{input}
</input>
`;

class CalcEquationParser extends BaseOutputParser {
  parse = async (output: string): Promise<string> => {
    try {
      if (/^<EOF\/>$/.test(output)) {
        return '';
      }

      return /<output>([\s\S]*)<\/output>/.exec(output)[1].trim();
    } catch (err) {
      return '';
    }
  };

  getFormatInstructions(): string {
    return `Provide your response in <output> block.`;
  }
}

const calcEquationPrompt = new PromptTemplate({
  template: CALC_EQUATION_GENERATE,
  inputVariables: ['input'],
  outputParser: new CalcEquationParser(),
});

const generateCalcEquationChain = new LLMChain({
  llm: gpt3Model,
  prompt: calcEquationPrompt,
  outputKey: 'data',
});

class GenericCalculator extends Tool {
  name = 'calculator';

  async _call(input: string) {
    const { data: generatedEquation } = await generateCalcEquationChain.call({
      input,
    });

    if (!generatedEquation) {
      return `Invalid equation. Please provide a valid equation. Try not to use word for numbers. For example, use 2 instead of two.`;
    }

    return new Calculator().call(generatedEquation);
  }

  description =
    'Useful for getting the result of a math expression. The input to this tool should be a valid mathmatical expression that could be executed by a simple calculator.';
}

This example uses all the modules we've covered in this post except Agents, plus the LLMChain chain module. Take a look at how LLMChain operates and how PromptTemplate and the LLM model work together.

In this post, I've introduced Langchain and several of its core modules, along with the foundational ideas for easily developing LLM-powered applications using this framework.

Langchain is still being developed at breakneck speed. There are application examples and many more modules I didn't cover here, including fascinating use cases like implementing long-term memory using VectorStore. If you're developing applications leveraging generative AI, definitely check it out. Thanks for reading.