This guide about Local LLMs was created 100% by AI. Please adjust based on your actual usage and requirements.
Why use local models?
Privacy
Your data never leaves your computer
No API costs
One-time setup, unlimited usage
Offline access
Work without internet connection
Full control
Customize model parameters
Requirements
Popular solutions
Ollama (Recommended for beginners)
Easy-to-use local model runner with simple CLI. Pros:- Simple installation
- Automatic model management
- Active community
- Optimized for Apple Silicon
- Llama 3.1 (8B, 70B)
- Qwen 2.5
- DeepSeek Coder
Ollama setup guide
Download and installation instructions
LM Studio
Desktop application with GUI for running local models. Pros:- User-friendly interface
- Model discovery and download
- Cross-platform (Mac, Windows, Linux)
- Built-in chat interface
- Users who prefer GUI over CLI
- Testing multiple models easily
- Quick model comparison
LM Studio
Download LM Studio
vLLM (Advanced)
High-performance inference engine for production deployments. Pros:- Fastest inference speed
- GPU optimization
- Production-ready
- API server included
- Technical users
- High-throughput needs
- Custom deployments
Connecting to SoloEnt
All local solutions expose an OpenAI-compatible API:1
Start local server
Launch your chosen solution (Ollama, LM Studio, etc.)
2
Configure in SoloEnt
Use OpenAI-compatible configuration:
3
Test connection
Send a test message to verify the setup.
Recommended models by use case
Writing & storytelling
Chinese content
Code & technical
Performance optimization
Use GPU acceleration
Use GPU acceleration
NVIDIA GPUs dramatically improve inference speed. Ensure CUDA is properly installed.
Choose appropriate model size
Choose appropriate model size
Larger models ≠ always better. 7B-13B models often provide the best speed/quality balance.
Quantization
Quantization
Use Q4 or Q5 quantized models to reduce memory usage with minimal quality loss.
Adjust context length
Adjust context length
Shorter context windows (4K-8K) run faster than long context (32K+).
Common issues
Model runs too slowly
Model runs too slowly
- Switch to smaller model (8B instead of 70B)
- Use quantized version (Q4_K_M)
- Enable GPU acceleration
- Reduce context window size
Out of memory
Out of memory
- Choose smaller model
- Close other applications
- Upgrade RAM
- Use higher quantization (Q3, Q4)
Connection refused
Connection refused
- Verify local server is running
- Check Base URL and port number
- Ensure no firewall blocking
- Try http://127.0.0.1 instead of localhost
Poor output quality
Poor output quality
- Try different prompt format
- Adjust temperature/top_p settings
- Switch to larger or different model
- Check if model is appropriate for your language
Next steps
API directory
Browse cloud API providers
Free API keys
Get free cloud API access