Chat with any LLM using Ollama
Title: Building a Chatbot with Ollama Chain Lit on Your Local System
Introduction
This blog is how to learn how to build a mini ChatGPT that runs on our machine using Mixtral, Ollama, llmlite, and Chainlit. And finally, we’ll add functionality that lets the chatbot answer questions on uploaded text files.
By leveraging your local LLM, you can build a chatbot without relying on cloud-based services, ensuring both privacy and efficiency. Let’s dive in!
Step 1: Install Ollama from the ollama website.
Before we start, you’ll need to have Python installed on your local system. If you haven’t already, download and install Python from the official Python website (https://www.python.org/downloads/).
This is a chat message with Ollama
Download ollama from ollama portal and run the following command
Ollama run Mistral or any LLM of your choice.
- First use Ollama to run LLM of you choice.
run ollama run xxxx
1. Setting up your python environment
python -m venv .venv
source .venv/bin/active
Copy the code below
Create a python and name it Chatapp.py
import chainlit as cl
import litellm
@cl.on_chat_start
def start_chat():
system_message = {
“role”: “system”,
“content”: “You are my helpful assistant, you are expert in many areas, please answer the questions correctly, if you don’t know the answer, please say I don’t know, please don’t make up answer. Please provide only correct answer.”
}
cl.user_session.set(“message_history”, [system_message])
@cl.on_message
async def on_message(message: cl.Message):
messages = cl.user_session.get(“message_history”)
if len(message.elements) > 0:
for element in message.elements:
with open(element.path, “r”) as uploaded_file:
content = uploaded_file.read()
messages.append({“role”: “user”, “content”: content})
confirm_message = cl.Message(content=f”Uploaded file: {element.name}”)
await confirm_message.send()
msg = cl.Message(content=””)
await msg.send()
messages.append({“role”: “user”, “content”: message.content})
response = await litellm.acompletion(
model=”ollama/mixtral”,
messages = messages,
api_base=”http://localhost:11434",
stream=True
)
async for chunk in response:
if chunk:
content = chunk.choices[0].delta.content
if content:
await msg.stream_token(content)
messages.append({“role”: “assistant”, “content”: msg.content})
await msg.update()
1. Install dependencies
pip install chainlit litellm
2. Run the application:
chainlit run ChatApp.py -w -h
3. Navigate to http://localhost:8000
Start interacting
