> ## Documentation Index
> Fetch the complete documentation index at: https://docs.soloent.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Local LLM Setup

> Run AI models on your own computer for privacy and offline access

<Note>
  This guide about Local LLMs was created 100% by AI. Please adjust based on your actual usage and requirements.
</Note>

## Why use local models?

<CardGroup cols={2}>
  <Card title="Privacy" icon="shield">
    Your data never leaves your computer
  </Card>

  <Card title="No API costs" icon="dollar-sign">
    One-time setup, unlimited usage
  </Card>

  <Card title="Offline access" icon="wifi-slash">
    Work without internet connection
  </Card>

  <Card title="Full control" icon="sliders">
    Customize model parameters
  </Card>
</CardGroup>

## Requirements

<Warning>
  Local models require significant hardware resources. Recommended minimum:

  * **RAM**: 16GB (32GB+ for larger models)
  * **Storage**: 10-50GB per model
  * **GPU**: Optional but highly recommended (NVIDIA with 8GB+ VRAM)
</Warning>

## Popular solutions

### Ollama (Recommended for beginners)

Easy-to-use local model runner with simple CLI.

**Pros:**

* Simple installation
* Automatic model management
* Active community
* Optimized for Apple Silicon

**Best models:**

* Llama 3.1 (8B, 70B)
* Qwen 2.5
* DeepSeek Coder

<Card title="Ollama setup guide" icon="rocket" href="https://ollama.ai">
  Download and installation instructions
</Card>

### LM Studio

Desktop application with GUI for running local models.

**Pros:**

* User-friendly interface
* Model discovery and download
* Cross-platform (Mac, Windows, Linux)
* Built-in chat interface

**Best for:**

* Users who prefer GUI over CLI
* Testing multiple models easily
* Quick model comparison

<Card title="LM Studio" icon="desktop" href="https://lmstudio.ai">
  Download LM Studio
</Card>

### vLLM (Advanced)

High-performance inference engine for production deployments.

**Pros:**

* Fastest inference speed
* GPU optimization
* Production-ready
* API server included

**Best for:**

* Technical users
* High-throughput needs
* Custom deployments

## Connecting to SoloEnt

All local solutions expose an OpenAI-compatible API:

<Steps>
  <Step title="Start local server">
    Launch your chosen solution (Ollama, LM Studio, etc.)
  </Step>

  <Step title="Configure in SoloEnt">
    Use OpenAI-compatible configuration:

    ```plaintext theme={null}
    Base URL: http://localhost:11434/v1  (Ollama default)
    API Key: ollama  (or leave blank)
    Model ID: llama3.1  (your model name)
    ```
  </Step>

  <Step title="Test connection">
    Send a test message to verify the setup.
  </Step>
</Steps>

## Recommended models by use case

### Writing & storytelling

| Model             | Size | RAM Required | Quality   |
| ----------------- | ---- | ------------ | --------- |
| **Llama 3.1 70B** | 40GB | 64GB+        | Excellent |
| **Qwen 2.5 32B**  | 20GB | 32GB+        | Very good |
| **Llama 3.1 8B**  | 5GB  | 16GB+        | Good      |

### Chinese content

| Model            | Size | RAM Required | Quality   |
| ---------------- | ---- | ------------ | --------- |
| **Qwen 2.5 72B** | 42GB | 64GB+        | Excellent |
| **GLM-4 9B**     | 6GB  | 16GB+        | Very good |
| **DeepSeek 67B** | 38GB | 64GB+        | Excellent |

### Code & technical

| Model                 | Size | RAM Required | Quality   |
| --------------------- | ---- | ------------ | --------- |
| **DeepSeek Coder V2** | 16GB | 32GB+        | Excellent |
| **CodeLlama 34B**     | 20GB | 32GB+        | Very good |
| **Qwen 2.5 Coder 7B** | 4GB  | 8GB+         | Good      |

## Performance optimization

<AccordionGroup>
  <Accordion title="Use GPU acceleration">
    NVIDIA GPUs dramatically improve inference speed. Ensure CUDA is properly installed.
  </Accordion>

  <Accordion title="Choose appropriate model size">
    Larger models ≠ always better. 7B-13B models often provide the best speed/quality balance.
  </Accordion>

  <Accordion title="Quantization">
    Use Q4 or Q5 quantized models to reduce memory usage with minimal quality loss.
  </Accordion>

  <Accordion title="Adjust context length">
    Shorter context windows (4K-8K) run faster than long context (32K+).
  </Accordion>
</AccordionGroup>

## Common issues

<AccordionGroup>
  <Accordion title="Model runs too slowly">
    * Switch to smaller model (8B instead of 70B)
    * Use quantized version (Q4\_K\_M)
    * Enable GPU acceleration
    * Reduce context window size
  </Accordion>

  <Accordion title="Out of memory">
    * Choose smaller model
    * Close other applications
    * Upgrade RAM
    * Use higher quantization (Q3, Q4)
  </Accordion>

  <Accordion title="Connection refused">
    * Verify local server is running
    * Check Base URL and port number
    * Ensure no firewall blocking
    * Try [http://127.0.0.1](http://127.0.0.1) instead of localhost
  </Accordion>

  <Accordion title="Poor output quality">
    * Try different prompt format
    * Adjust temperature/top\_p settings
    * Switch to larger or different model
    * Check if model is appropriate for your language
  </Accordion>
</AccordionGroup>

## Next steps

<CardGroup cols={2}>
  <Card title="API directory" icon="book-open" href="./api-directory">
    Browse cloud API providers
  </Card>

  <Card title="Free API keys" icon="gift" href="./free-api-keys">
    Get free cloud API access
  </Card>
</CardGroup>