Nvidia Unveils Futuristic Gaming Experience at Computex 2023
News

Nvidia Unveils Futuristic Gaming Experience at Computex 2023, Blending Gaming and AI

by Isabel
May 29, 2023
0

At Computex 2023, Nvidia displays a futuristic gaming experience that...

Read more
Adobe Introduces Powerful Generative AI Tools in Photoshop

Adobe Introduces Powerful Generative AI Tools in Photoshop Beta

May 29, 2023
Adobe Photoshop's Generative Fill Feature

Exploring the Power of Adobe Photoshop’s Generative Fill Feature

May 27, 2023
NVIDIA and Microsoft Partner to Accelerate AI

NVIDIA and Microsoft Partner to Accelerate AI

May 25, 2023
google photos security and privacy

Exploring the Top 5 Privacy and Security Risks of using Google Photos

May 24, 2023
AutoGPT in Linux

How to download and install Auto GPT in Linux

May 4, 2023
Smol Developer The AI That Can Code for You

Smol Developer: The AI That Can Code for You

May 18, 2023
ChatGPT login

ChatGPT Login Guide to Free OpenAI Access

May 20, 2023
How to use Auto GPT and Agent GPT

How to Use Auto GPT and Agent GPT

May 4, 2023
everything you need to know about Adobe firefly

Everything You Need to Know About Adobe Firefly

May 29, 2023
Voice Changer AI

Voice Changer AI: The Future of Communication

May 24, 2023
Cloudbooklet
  • News
  • Artificial Intelligence
  • Linux
  • Google Cloud
  • AWS
No Result
View All Result
Cloudbooklet
  • News
  • Artificial Intelligence
  • Linux
  • Google Cloud
  • AWS
No Result
View All Result
Cloudbooklet
No Result
View All Result
Home Artificial Intelligence

LocalAI: A Drop-In Replacement for OpenAI’s REST API

by Isabel
May 17, 2023
in Artificial Intelligence
Reading Time: 6 mins read
LocalAI Drop-In Replacement for OpenAI's REST API
Share on FacebookShare on TwitterShare on WhatsAppShare on Telegram

LocalAI is a drop-in replacement REST API for local inference that is consistent with OpenAI API requirements. It enables models to be run locally or on-prem using consumer-grade hardware and supports different model families that are compatible with the ggml format.

Table of Contents

  1. Usage
  2. Benefits of using Local AI
  3. Features of LocalAI

LocalAI: A Drop-In Replacement for OpenAI’s REST API

You might also like

ChatGPT app

The Easiest Way to Download ChatGPT App Free

May 31, 2023
LLM Connected with APIs

Gorilla: LLM Connected with APIs

May 31, 2023

LocalAI is a drop-in replacement REST API for local inference that is consistent with OpenAI API requirements. It enables models to be run locally or on-prem using consumer-grade hardware and supports different model families that are compatible with the ggml format.

LocalAI

Local AI is still in its early stages, but it has the potential to be a useful tool for developers and researchers that need to run AI models locally.

Usage

LocalAI comes by default as a container image.

The simplest approach to start LocalAI is using docker-compose:


git clone https://github.com/go-skynet/LocalAI

cd LocalAI

# (optional) Checkout a specific LocalAI tag
# git checkout -b build <TAG>

# copy your models to models/
cp your-model.bin models/

# (optional) Edit the .env file to set things like context size and threads
# vim .env

# start with docker-compose
docker-compose up -d --pull always
# or you can build the images with:
# docker-compose up -d --build

# Now API is accessible at localhost:8080
curl http://localhost:8080/v1/models
# {"object":"list","data":[{"id":"your-model.bin","object":"model"}]}

curl http://localhost:8080/v1/completions -H "Content-Type: application/json" -d '{
     "model": "your-model.bin",            
     "prompt": "A long time ago in a galaxy far, far away",
     "temperature": 0.7
   }'

Docker

Example of starting the API with `docker`:

docker run -p 8080:8080 -ti --rm quay.io/go-skynet/local-ai:latest --models-path /path/to/models --context-size 700 --threads 4

Build locally:

In order to build the Local AI container image locally you can use docker:

# build the image
docker build -t LocalAI .
docker run LocalAI

Or you can build the binary with make:

make build

Build on mac

Building on Mac (M1 or M2) works, but you may need to install some prerequisites using brew.

The below has been tested by one mac user and found to work. Note that this doesn’t use docker to run the server:

# install build dependencies
brew install cmake
brew install go

# clone the repo
git clone https://github.com/go-skynet/LocalAI.git

cd LocalAI

# build the binary
make build

# Download gpt4all-j to models/
wget https://gpt4all.io/models/ggml-gpt4all-j.bin -O models/ggml-gpt4all-j

# Use a template from the examples
cp -rf prompt-templates/ggml-gpt4all-j.tmpl models/

# Run LocalAI
./local-ai --models-path ./models/ --debug

# Now API is accessible at localhost:8080
curl http://localhost:8080/v1/models

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
     "model": "ggml-gpt4all-j",
     "messages": [{"role": "user", "content": "How are you?"}],
     "temperature": 0.9 
   }'

Run LocalAI in Kubernetes

LocalAI can be installed inside Kubernetes with helm.

Add the helm repo:

helm repo add go-skynet https://go-skynet.github.io/helm-charts/

Create a values.yaml file with your settings:

deployment:
  image: quay.io/go-skynet/local-ai:latest
  env:
    threads: 4
    contextSize: 1024
    modelsPath: "/models"
# Optionally create a PVC, mount the PV to the LocalAI Deployment,
# and download a model to prepopulate the models directory
modelsVolume:
  enabled: true
  url: "https://gpt4all.io/models/ggml-gpt4all-j.bin"
  pvc:
    size: 6Gi
    accessModes:
    - ReadWriteOnce
  auth:
    # Optional value for HTTP basic access authentication header
    basic: "" # 'username:password' base64 encoded
service:
  type: ClusterIP
  annotations: {}
  # If using an AWS load balancer, you'll need to override the default 60s load balancer idle timeout
  # service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "1200"

Install the helm chart:

helm repo update
helm install local-ai go-skynet/local-ai -f values.yaml

Benefits of using Local AI

  • Reduced latency: By removing the requirement to make queries to a distant server, Local AI can minimize latency. This is particularly useful for applications requiring real-time reactions, such as gaming and robotics.
  • Improving privacy: By storing data on the local device, Local AI can increase privacy. This is particularly useful for applications dealing with sensitive data, such as healthcare and banking.
  • Saving money: By removing the need to pay for cloud computing resources, Local AI can save money. This is especially useful for apps that are only used occasionally or need to analyze tiny quantities of data.

If you want to run AI models locally, Local AI is an excellent choice. It is a useful tool for improving the performance, security, and cost-effectiveness of your applications.

Also Read e2b: Create your Virtual Software Developer using AI.

Features of LocalAI

  • Simple to use: LocalAI is simple to use, even for novices. The documentation is straightforward and concise, and there is a strong user community eager to assist.
  • Powerful:  LocalAI is an extremely strong tool that may be used to create complicated AI applications. It is still in the works, but it has the potential to change the way AI is built.
  • Flexible:  Local AI is adaptable and can be used to construct AI applications in a wide range of languages and frameworks.

Local AI is an excellent choice if you need a strong and adaptable tool to run AI models locally. It is simple to use and has a huge number of users that are eager to assist.

This article is to help you learn Local AI. We trust that it has been helpful to you. Please feel free to share your thoughts and feedback in the comment section below.

Continue Reading
Share1Tweet1SendShare
Isabel

Isabel

Help us grow and support our blog! Your contribution can make a real difference in providing valuable content to our readers. Join us in our journey by supporting our blog today!
Buy me a Coffee

Related Posts

Soundstorm-Pytorch

Soundstorm-Pytorch: A Powerful Tool for Audio Generation

May 30, 2023
Midjourney vs Adobe Firefly

Midjourney vs Adobe Firefly: A Comparison of Two AI Image Generation Tools

May 30, 2023
ChatGPT

How to Use ChatGPT Code Interpreter

May 31, 2023
Leonardo AI Login

How to login and use Leonardo AI to generate high-quality image

May 30, 2023

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

I agree to the Terms & Conditions and Privacy Policy.

Cloudbooklet

Welcome to our technology blog, where we explore the latest advancements in the field of artificial intelligence (AI) and how they are revolutionizing cloud computing. In this blog, we dive into the powerful capabilities of cloud platforms like Google Cloud Platform (GCP), Amazon Web Services (AWS), and Microsoft Azure, and how they are accelerating the adoption and deployment of AI solutions across various industries. Join us on this exciting journey as we explore the endless possibilities of AI and cloud computing.

  • About
  • Contact
  • Disclaimer
  • Privacy Policy

Cloudbooklet © 2023 All rights reserved.

No Result
View All Result
  • News
  • Artificial Intelligence
  • Linux
  • Google Cloud
  • AWS

Cloudbooklet © 2023 All rights reserved.

This website uses cookies. By continuing to use this website you are giving consent to cookies being used. Visit our Privacy and Cookie Policy.