Ollama

YanFlow cluster integrates Ollama, an open-source large language model (LLM) platform designed to enable users to easily run, manage, and interact with large language models locally. It provides a rich library of pre-trained models, supporting various natural language processing tasks such as text generation, translation, and code writing.

Starting the Service

Start the large model service in YanFlow cluster.

After successful startup, the API and WebUI ports will be mapped respectively.

Click WebUI to view how to use it.

How to Use Offline Large Models?

Integrating with Development Tools

To integrate offline large models with development tools, please refer to the following documentation for configuration.

Note: You need to configure the IP:PORT information obtained from the image above into your development tools to ensure correct connection.

Development Tool	Configuration Guide
VS Code	Click to Configure (Ollama Integration)
JetBrains (IntelliJ, PyCharm, WebStorm, etc.)	Click to Configure (Ollama Integration)

API Integration

If you need to access offline large models through API, you can achieve this by configuring environment variables or specifying the model's address and port in your code.

Core prerequisite: You must configure the IP:PORT obtained from the image above (i.e., the address where the model service is running) as the target address for API calls.

Typical API Call Example (Python)

Most client libraries that interact with large models (such as Ollama client or OpenAI-compatible clients) allow you to connect to your local service by specifying base_url.

import openai  # Assuming use of OpenAI-compatible client

# Replace with your actual IP and PORT
OLLAMA_API_BASE_URL = "http://your-IP:port/v1" 

client = openai.OpenAI(
    base_url=OLLAMA_API_BASE_URL,
    api_key="sk-...",  # Offline services typically don't require API Key, but some clients need a placeholder
)

response = client.chat.completions.create(
    model="model-name",  # Replace with your actual deployed model name
    messages=[
        {"role": "user", "content": "Please write a quicksort algorithm in Python."},
    ],
)

print(response.choices[0].message.content)

Configuring with Environment Variables

For convenience when calling from terminal or scripts, you can set the model service address as environment variable OLLAMA_HOST or OPENAI_API_BASE (depending on the client you use):

# Assuming your model service is running on port 11434 of 192.168.1.100

# Method 1: Set OLLAMA_HOST
export OLLAMA_HOST=192.168.1.100:11434

# Method 2: Set OPENAI API-compatible Base URL
export OPENAI_API_BASE="http://192.168.1.100:11434/v1"

After configuration, use the corresponding client tools or code to automatically connect to your specified local service.