Nvidia Unveils Futuristic Gaming Experience at Computex 2023
News

Nvidia Unveils Futuristic Gaming Experience at Computex 2023, Blending Gaming and AI

by Isabel
May 29, 2023
0

At Computex 2023, Nvidia displays a futuristic gaming experience that...

Read more
Adobe Introduces Powerful Generative AI Tools in Photoshop

Adobe Introduces Powerful Generative AI Tools in Photoshop Beta

May 29, 2023
Adobe Photoshop's Generative Fill Feature

Exploring the Power of Adobe Photoshop’s Generative Fill Feature

May 27, 2023
NVIDIA and Microsoft Partner to Accelerate AI

NVIDIA and Microsoft Partner to Accelerate AI

May 25, 2023
google photos security and privacy

Exploring the Top 5 Privacy and Security Risks of using Google Photos

May 24, 2023
Marvin AI A Powerful Tool for Building AI-Powered Software

Marvin AI: A Powerful Tool for Building AI-Powered Software

May 18, 2023
AgentGPT

How to install AgentGPT

May 10, 2023
How to use Autogpt in web browser

How to use AutoGPT in web browser

May 4, 2023
What is ChoasGPT

What is chaosGPT?

April 20, 2023
AgentLLM

AgentLLM (Large Language Model)

April 28, 2023
Building an Effective ML Deployment Stack with Docker on Ubuntu 22.04

Building an Effective ML Deployment Stack with Docker on Ubuntu 22.04

February 28, 2023
Cloudbooklet
  • News
  • Artificial Intelligence
  • Linux
  • Google Cloud
  • AWS
No Result
View All Result
Cloudbooklet
  • News
  • Artificial Intelligence
  • Linux
  • Google Cloud
  • AWS
No Result
View All Result
Cloudbooklet
No Result
View All Result
Home Artificial Intelligence

FastChat: A New Platform for Chatbot Development and Evaluation

by Cloudbooklet
May 8, 2023
in Artificial Intelligence
Reading Time: 11 mins read
Fastchat
Share on FacebookShare on TwitterShare on WhatsAppShare on Telegram

FastChat is a platform that seeks to simplify and speed up the process of developing and evaluating chatbots. With chatbots gaining popularity and usefulness in areas like customer service, education, entertainment, and health care, creating and testing chatbots can be a challenging and resource-intensive task. In this article, we’ll introduce you to FastChat, a new platform that streamlines the chatbot development and evaluation process.

Table of Contents

  1. Installation
  2. Model Weights
    1. Vicuna Weights
    2. Vicuna-7B
    3. Vicuna-13B
  3. Old weights
  4. Generate API
  5. Evaluation

    FastChat provides a user-friendly interface that allows users to create, test, and deploy chatbots in minutes. FastChat also offers a rich set of features, such as natural language understanding, dialogue management, response generation, and analytics. FastChat enables users to build chatbots for different purposes and scenarios, such as conversational agents, question answering systems, task-oriented bots, and social chatbots. FastChat supports multiple languages and platforms, such as web, mobile, and voice. FastChat is designed to help users create high-quality chatbots that can engage and satisfy their target audiences.

    You might also like

    ChatGPT app

    The Easiest Way to Download ChatGPT App Free

    May 31, 2023
    LLM Connected with APIs

    Gorilla: LLM Connected with APIs

    May 31, 2023

    Installation

    Method 1: Install With pip

    pip3 install fschat

    Method 2: Get source from GitHub

    1. Clone this repository and navigate to the FastChat folder.
    git clone https://github.com/lm-sys/FastChat.git
    cd FastChat

    If you are running on Mac:

    brew install rust cmake

    2. Install Package

    pip3 install --upgrade pip  # enable PEP 660 support
    pip3 install -e .

    Model Weights

    Vicuna Weights

    To comply with the LLaMA model licence, they provide Vicuna weights as delta weights. To acquire the Vicuna weights, add delta to the original LLaMA weights.

    Instructions:

    1. Get the original LLaMA weights in the huggingface format by following the instructions here.
    2. Apply our delta to the scripts below to acquire Vicuna weights. They will download delta weights from our Hugging Face account automatically.

    Weights v1.1 are only compatible with transformers>=4.28.0 and fschat >= 0.2.0. Please update your local packages as needed. If you use the above instructions to perform a clean install, you should obtain all of the right versions.

    Vicuna-7B

    This conversion command requires around 30 GB of CPU RAM. If you don’t have enough memory, see the “Low CPU Memory Conversion” section below.

    python3 -m fastchat.model.apply_delta \
        --base-model-path /path/to/llama-7b \
        --target-model-path /output/path/to/vicuna-7b \
        --delta-path lmsys/vicuna-7b-delta-v1.1

    Vicuna-13B

    This conversion command needs around 60 GB of CPU RAM. See the “Low CPU Memory Conversion” section below if you do not have enough memory.

    python3 -m fastchat.model.apply_delta \
        --base-model-path /path/to/llama-13b \
        --target-model-path /output/path/to/vicuna-13b \
        --delta-path lmsys/vicuna-13b-delta-v1.1

    Old weights

    Low CPU Memory Conversion

    You can try these methods to reduce the CPU RAM requirement of weight conversion.

    1. Add --low-cpu-mem to the preceding commands to break huge files into smaller ones and use the disc as temporary storage. This can keep the maximum RAM below 16GB.
    2. Make a huge swap file and rely on the operating system to use the disc as virtual memory automatically.

    FastChat-T5

    Simply run the line below to start chatting. It will automatically download the weights from a Hugging Face repo.

    python3 -m fastchat.serve.cli --model-path lmsys/fastchat-t5-3b-v1.0

    Supported Models

    The following models are tested:

    • Vicuna, Alpaca, LLaMA, Koala
    • lmsys/fastchat-t5-3b-v1.0
    • BlinkDL/RWKV-4-Raven
    • databricks/dolly-v2-12b
    • OpenAssistant/oasst-sft-1-pythia-12b
    • project-baize/baize-lora-7B
    • StabilityAI/stablelm-tuned-alpha-7b
    • THUDM/chatglm-6b

    Single GPU

    The following command takes around 28GB of GPU memory for Vicuna-13B and 14GB of GPU memory for Vicuna-7B. If you don’t have enough memory, see the “No Enough Memory” section below.

    python3 -m fastchat.serve.cli --model-path /path/to/model/weights

    Multiple GPUs

    Model parallelism can be used to aggregate GPU memory from numerous GPUs on the same system.

    python3 -m fastchat.serve.cli --model-path /path/to/model/weights --num-gpus 2

    CPU Only

    This works just on the CPU and does not require a GPU. Vicuna-13B requires around 60GB of CPU memory, whereas Vicuna-7B requires approximately 30GB of CPU memory.

    python3 -m fastchat.serve.cli --model-path /path/to/model/weights --device cpu

    Metal Backend (Mac Computers with Apple Silicon or AMD GPUs)

    Use --device mps to enable GPU acceleration on Mac computers (requires torch >= 2.0). Use --load-8bit to turn on 8-bit compression.

    python3 -m fastchat.serve.cli --model-path /path/to/model/weights --device mps --load-8bit

    Vicuna-7B can run on a 32GB M1 Macbook with 1 – 2 words / second.

    Not Enough Memory

    If you don’t have enough RAM, you may use 8-bit compression by adding –load-8bit to the preceding instructions. With somewhat reduced model quality, this can cut memory consumption in half. It works with the CPU, GPU, and Metal backends. Vicuna-13B can operate on a single NVIDIA 3090/4080/T4/V100(16GB) GPU with 8-bit compression.

    python3 -m fastchat.serve.cli --model-path /path/to/model/weights --load-8bit

    Additionally, you may use --cpu-offloading to the aforementioned instructions to unload weights that do not fit on your GPU to CPU memory. This necessitates the activation of 8-bit compression and the installation of the bitsandbytes package, which is only accessible on Linux operating systems.

    More Platforms

    • MLC LLM, backed by TVM Unity compiler, deploys Vicuna natively on phones, consumer-class GPUs and web browsers via Vulkan, Metal, CUDA and WebGPU.

    Serving with Web GUI

    FastChat

    You’ll need three major components to serve utilizing the web UI: web servers that interact with users, model workers that host one or more models, and a controller to synchronize the webserver and model workers. The following commands should be entered into your terminal:

    Launch the controller

    python3 -m fastchat.serve.controller

    This controller manages the distributed workers.

    Launch the model worker

    python3 -m fastchat.serve.model_worker --model-path /path/to/model/weights

    Wait until the model loading procedure is complete and you notice “Uvicorn running on…”. You can launch numerous model workers at the same time to service various models. The model worker will immediately connect to the controller.

    Send a test message using the following command to confirm that your model worker is correctly linked to your controller:

    python3 -m fastchat.serve.test_message --model-name vicuna-7b

    You will see a short output.

    Launch the Gradio web server

    python3 -m fastchat.serve.gradio_web_server

    This is the user interface with which users will interact.

    You will be able to serve your models via the web UI if you follow these instructions. You may now open your browser and start chatting with a model.

    Generate API

    Huggingface Generation APIs

    See fastchat/serve/huggingface_api.py

    OpenAI-compatible RESTful APIs & SDK

    See docs/openai_api.md

    Evaluation

    GPT-4 is the foundation of our AI-enhanced assessment workflow. This section gives a high-level overview of the pipeline. Please see the evaluation documents for further information.

    Pipeline Steps

    1. Create replies using several models: For ChatGPT, use qa_baseline_gpt35.pyor provide the model checkpoint and run get_model_answer.py for Vicuna and other models.
    2. Generate reviews with GPT-4: GPT-4 may be used to generate reviews automatically. If you do not have access to the GPT-4 API, you can do this step manually.
    3. Generate visualization data: Run generate_webpage_data_from_table.py to create data for a static website that will allow you to visualize the evaluation data.
    4. Data visualization: Under the webpage directory, provide a static website. To serve the webpage locally, use python3 -m http.server

    Data Format and Contribution

    For evaluation, they employ a data format encoded with JSON lines. Models, prompts, reviewers, questions, replies, and reviews are all included in the structure.

    By gaining access to the appropriate data, you may customize the evaluation process or contribute to the data.

    For detailed instructions, please refer to the evaluation documentation.

    Also Read ChatGPT4 for Free.

    This article helps you learn about FastChat. We trust that it has been helpful to you. Please feel free to share your thoughts and feedback in the comment section below.

    Share1Tweet1SendShare
    Cloudbooklet

    Cloudbooklet

    Help us grow and support our blog! Your contribution can make a real difference in providing valuable content to our readers. Join us in our journey by supporting our blog today!
    Buy me a Coffee

    Related Posts

    Soundstorm-Pytorch

    Soundstorm-Pytorch: A Powerful Tool for Audio Generation

    May 30, 2023
    Midjourney vs Adobe Firefly

    Midjourney vs Adobe Firefly: A Comparison of Two AI Image Generation Tools

    May 30, 2023
    ChatGPT

    How to Use ChatGPT Code Interpreter

    May 31, 2023
    Leonardo AI Login

    How to login and use Leonardo AI to generate high-quality image

    May 30, 2023

    Leave a Reply Cancel reply

    Your email address will not be published. Required fields are marked *

    I agree to the Terms & Conditions and Privacy Policy.

    • Trending
    • Comments
    • Latest
    DragGAN The AI-Powered Image Editing Tool

    DragGAN: The AI-Powered Image Editing Tool That Makes Editing Images Easy

    May 30, 2023
    DragGAN AI editing Tool Install and Use DragGAN Photo Editor

    DragGAN AI editing Tool Install and Use DragGAN Photo Editor

    May 27, 2023
    Bard API key

    Everything You Need to Know About Google’s Bard API Key

    May 20, 2023
    Install PHP 8.1 on Ubuntu

    How to Install or Upgrade PHP 8.1 on Ubuntu 20.04

    May 17, 2023
    DragGAN The AI-Powered Image Editing Tool

    DragGAN: The AI-Powered Image Editing Tool That Makes Editing Images Easy

    75
    Upgrade PHP version to PHP 7.4 on Ubuntu

    Upgrade PHP version to PHP 7.4 on Ubuntu

    28
    Install Odoo 13 on Ubuntu 18.04 with Nginx - Google Cloud

    Install Odoo 13 on Ubuntu 18.04 with Nginx – Google Cloud

    25
    Best Performance WordPress with Google Cloud CDN and Load Balancing

    Best Performance WordPress with Google Cloud CDN and Load Balancing

    23
    How to Setup SSH Keys on Ubuntu

    How to Setup SSH Keys on Ubuntu 20.04

    May 31, 2023
    ChatGPT app

    The Easiest Way to Download ChatGPT App Free

    May 31, 2023
    LLM Connected with APIs

    Gorilla: LLM Connected with APIs

    May 31, 2023
    Soundstorm-Pytorch

    Soundstorm-Pytorch: A Powerful Tool for Audio Generation

    May 30, 2023

    Popular Articles

    • DragGAN The AI-Powered Image Editing Tool

      DragGAN: The AI-Powered Image Editing Tool That Makes Editing Images Easy

      1444 shares
      Share 578 Tweet 361
    • DragGAN AI editing Tool Install and Use DragGAN Photo Editor

      339 shares
      Share 136 Tweet 85
    • Auto-Photoshop-Stable Diffusion-Plugin: A New Way to Create AI-Generated Images in Photoshop

      70 shares
      Share 28 Tweet 18
    • InternGPT: A New Way to Interact with ChatGPT

      54 shares
      Share 22 Tweet 14
    • Midjourney vs Adobe Firefly: A Comparison of Two AI Image Generation Tools

      11 shares
      Share 4 Tweet 3
    Cloudbooklet

    Welcome to our technology blog, where we explore the latest advancements in the field of artificial intelligence (AI) and how they are revolutionizing cloud computing. In this blog, we dive into the powerful capabilities of cloud platforms like Google Cloud Platform (GCP), Amazon Web Services (AWS), and Microsoft Azure, and how they are accelerating the adoption and deployment of AI solutions across various industries. Join us on this exciting journey as we explore the endless possibilities of AI and cloud computing.

    • About
    • Contact
    • Disclaimer
    • Privacy Policy

    Cloudbooklet © 2023 All rights reserved.

    No Result
    View All Result
    • News
    • Artificial Intelligence
    • Linux
    • Google Cloud
    • AWS

    Cloudbooklet © 2023 All rights reserved.

    This website uses cookies. By continuing to use this website you are giving consent to cookies being used. Visit our Privacy and Cookie Policy.