Skip to content

ONNX Runtime Setup (Advanced)

Use this path when you want embedding.provider: onnx.

Pali's ONNX provider uses github.com/yalue/onnxruntime_go, which requires a platform-native ONNX Runtime shared library at runtime.

Requirements

  1. ONNX model files:
  2. models/all-MiniLM-L6-v2/model.onnx
  3. models/all-MiniLM-L6-v2/tokenizer.json
  4. ONNX Runtime shared library (.dylib, .so, or .dll)

macOS

  1. Install ONNX Runtime:
  2. brew install onnxruntime
  3. Formula reference: https://formulae.brew.sh/formula/onnxruntime
  4. Verify library path:
  5. usually /opt/homebrew/lib/libonnxruntime.dylib on Apple Silicon
  6. If needed, set:
  7. export ONNXRUNTIME_SHARED_LIBRARY_PATH=/opt/homebrew/lib/libonnxruntime.dylib

Windows

  1. Download ONNX Runtime release package (C/C++ binaries):
  2. https://github.com/microsoft/onnxruntime/releases
  3. Extract and locate onnxruntime.dll.
  4. Set environment variable:
  5. setx ONNXRUNTIME_SHARED_LIBRARY_PATH "C:\\path\\to\\onnxruntime.dll"
  6. Install Microsoft Visual C++ 2019 runtime if missing:
  7. requirement noted by onnxruntime_go: https://github.com/yalue/onnxruntime_go/blob/master/README.md

Linux

  1. Download ONNX Runtime package from releases:
  2. https://github.com/microsoft/onnxruntime/releases
  3. Locate libonnxruntime.so and set:
  4. export ONNXRUNTIME_SHARED_LIBRARY_PATH=/path/to/libonnxruntime.so

Config Example

embedding:
  provider: onnx
  model_path: ./models/all-MiniLM-L6-v2/model.onnx
  tokenizer_path: ./models/all-MiniLM-L6-v2/tokenizer.json

Troubleshooting

  • Error: Error loading ONNX shared library ...
  • shared library path is missing/wrong
  • set ONNXRUNTIME_SHARED_LIBRARY_PATH explicitly
  • Error: model file not found
  • run pali init -download-model to fetch missing model artifacts

References