EssentialAI/rnj-1-instruct-GGUF

This is a GGUF-formatted checkpoint of rnj-1-instruct suitable for use in llama.cpp. This has been quantized with the Q4_K_M scheme, which results in model weights of size 4.8GB.

Support for this model has been PR'd upstream, but in the meantime you can build from source with these instructions for MacOS. For Linux, install cmake using your package manager. For Windows, consult the llama.cpp build guide.

brew install cmake
git clone https://github.com/Essential-AI/llama.cpp
cd llama.cpp
git checkout rnj-1
cmake -B build
cmake --build build --config Release

To run this model in a web UI, use this command and open your browser to http://localhost:8080:

build/bin/llama-server -hf EssentialAI/rnj-1-instruct-GGUF -c 0

To run it in the CLI, use this command:

build/bin/llama-cli -hf EssentialAI/rnj-1-instruct-GGUF