Cloudbooklet
  • News
  • Artificial Intelligence
  • Applications
  • Linux
No Result
View All Result
Cloudbooklet
  • News
  • Artificial Intelligence
  • Applications
  • Linux
No Result
View All Result
Cloudbooklet
No Result
View All Result
Home Artificial Intelligence

FastChat: A New Platform for Chatbot Development and Evaluation

by Cloudbooklet
5 months ago
in Artificial Intelligence
Fastchat
ShareTweetSendShare
Readers like you help support Cloudbooklet. When you make a purchase using links on our site, we may earn an affiliate commission.

FastChat is a new chatbot development and evaluation platform that seeks to provide a standardized and scalable method of generating and testing conversational bots.

ADVERTISEMENT

FastChat is a platform that seeks to simplify and speed up the process of developing and evaluating chatbots. With chatbots gaining popularity and usefulness in areas like customer service, education, entertainment, and health care, creating and testing chatbots can be a challenging and resource-intensive task. In this article, we’ll introduce you to FastChat, a new platform that streamlines the chatbot development and evaluation process.

Table of Contents

  1. Installation
  2. Model Weights
    1. Vicuna Weights
    2. Vicuna-7B
    3. Vicuna-13B
  3. Old weights
  4. Generate API
  5. Evaluation

    FastChat provides a user-friendly interface that allows users to create, test, and deploy chatbots in minutes. FastChat also offers a rich set of features, such as natural language understanding, dialogue management, response generation, and analytics. FastChat enables users to build chatbots for different purposes and scenarios, such as conversational agents, question answering systems, task-oriented bots, and social chatbots. FastChat supports multiple languages and platforms, such as web, mobile, and voice. FastChat is designed to help users create high-quality chatbots that can engage and satisfy their target audiences.

    Installation

    Method 1: Install With pip

    ADVERTISEMENT
    pip3 install fschat

    Method 2: Get source from GitHub

    You might also like

    Google Bard Extension

    Google Bard Extensions: How to Link Your Gmail, Docs, Maps, and More to an AI Chatbot

    8 hours ago
    Validator Ai

    Validator AI: The AI Powered Business Idea Validator

    1 day ago
    1. Clone this repository and navigate to the FastChat folder.
    git clone https://github.com/lm-sys/FastChat.git
    cd FastChat

    If you are running on Mac:

    brew install rust cmake

    2. Install Package

    ADVERTISEMENT
    pip3 install --upgrade pip  # enable PEP 660 support
    pip3 install -e .

    Model Weights

    Vicuna Weights

    To comply with the LLaMA model licence, they provide Vicuna weights as delta weights. To acquire the Vicuna weights, add delta to the original LLaMA weights.

    Instructions:

    ADVERTISEMENT
    1. Get the original LLaMA weights in the huggingface format by following the instructions here.
    2. Apply our delta to the scripts below to acquire Vicuna weights. They will download delta weights from our Hugging Face account automatically.

    Weights v1.1 are only compatible with transformers>=4.28.0 and fschat >= 0.2.0. Please update your local packages as needed. If you use the above instructions to perform a clean install, you should obtain all of the right versions.

    Vicuna-7B

    This conversion command requires around 30 GB of CPU RAM. If you don’t have enough memory, see the “Low CPU Memory Conversion” section below.

    ADVERTISEMENT
    python3 -m fastchat.model.apply_delta \
        --base-model-path /path/to/llama-7b \
        --target-model-path /output/path/to/vicuna-7b \
        --delta-path lmsys/vicuna-7b-delta-v1.1

    Vicuna-13B

    This conversion command needs around 60 GB of CPU RAM. See the “Low CPU Memory Conversion” section below if you do not have enough memory.

    python3 -m fastchat.model.apply_delta \
        --base-model-path /path/to/llama-13b \
        --target-model-path /output/path/to/vicuna-13b \
        --delta-path lmsys/vicuna-13b-delta-v1.1

    Old weights

    Low CPU Memory Conversion

    ADVERTISEMENT

    You can try these methods to reduce the CPU RAM requirement of weight conversion.

    1. Add --low-cpu-mem to the preceding commands to break huge files into smaller ones and use the disc as temporary storage. This can keep the maximum RAM below 16GB.
    2. Make a huge swap file and rely on the operating system to use the disc as virtual memory automatically.

    FastChat-T5

    Simply run the line below to start chatting. It will automatically download the weights from a Hugging Face repo.

    python3 -m fastchat.serve.cli --model-path lmsys/fastchat-t5-3b-v1.0

    Supported Models

    The following models are tested:

    • Vicuna, Alpaca, LLaMA, Koala
    • lmsys/fastchat-t5-3b-v1.0
    • BlinkDL/RWKV-4-Raven
    • databricks/dolly-v2-12b
    • OpenAssistant/oasst-sft-1-pythia-12b
    • project-baize/baize-lora-7B
    • StabilityAI/stablelm-tuned-alpha-7b
    • THUDM/chatglm-6b

    Single GPU

    The following command takes around 28GB of GPU memory for Vicuna-13B and 14GB of GPU memory for Vicuna-7B. If you don’t have enough memory, see the “No Enough Memory” section below.

    python3 -m fastchat.serve.cli --model-path /path/to/model/weights

    Multiple GPUs

    Model parallelism can be used to aggregate GPU memory from numerous GPUs on the same system.

    python3 -m fastchat.serve.cli --model-path /path/to/model/weights --num-gpus 2

    CPU Only

    This works just on the CPU and does not require a GPU. Vicuna-13B requires around 60GB of CPU memory, whereas Vicuna-7B requires approximately 30GB of CPU memory.

    python3 -m fastchat.serve.cli --model-path /path/to/model/weights --device cpu

    Metal Backend (Mac Computers with Apple Silicon or AMD GPUs)

    Use --device mps to enable GPU acceleration on Mac computers (requires torch >= 2.0). Use --load-8bit to turn on 8-bit compression.

    python3 -m fastchat.serve.cli --model-path /path/to/model/weights --device mps --load-8bit

    Vicuna-7B can run on a 32GB M1 Macbook with 1 – 2 words / second.

    Not Enough Memory

    If you don’t have enough RAM, you may use 8-bit compression by adding –load-8bit to the preceding instructions. With somewhat reduced model quality, this can cut memory consumption in half. It works with the CPU, GPU, and Metal backends. Vicuna-13B can operate on a single NVIDIA 3090/4080/T4/V100(16GB) GPU with 8-bit compression.

    python3 -m fastchat.serve.cli --model-path /path/to/model/weights --load-8bit

    Additionally, you may use --cpu-offloading to the aforementioned instructions to unload weights that do not fit on your GPU to CPU memory. This necessitates the activation of 8-bit compression and the installation of the bitsandbytes package, which is only accessible on Linux operating systems.

    More Platforms

    • MLC LLM, backed by TVM Unity compiler, deploys Vicuna natively on phones, consumer-class GPUs and web browsers via Vulkan, Metal, CUDA and WebGPU.

    Serving with Web GUI

    Fastchat
    FastChat: A New Platform for Chatbot Development and Evaluation 1

    You’ll need three major components to serve utilizing the web UI: web servers that interact with users, model workers that host one or more models, and a controller to synchronize the webserver and model workers. The following commands should be entered into your terminal:

    Launch the controller

    python3 -m fastchat.serve.controller

    This controller manages the distributed workers.

    Launch the model worker

    python3 -m fastchat.serve.model_worker --model-path /path/to/model/weights

    Wait until the model loading procedure is complete and you notice “Uvicorn running on…”. You can launch numerous model workers at the same time to service various models. The model worker will immediately connect to the controller.

    Send a test message using the following command to confirm that your model worker is correctly linked to your controller:

    python3 -m fastchat.serve.test_message --model-name vicuna-7b

    You will see a short output.

    Launch the Gradio web server

    python3 -m fastchat.serve.gradio_web_server

    This is the user interface with which users will interact.

    You will be able to serve your models via the web UI if you follow these instructions. You may now open your browser and start chatting with a model.

    Generate API

    Huggingface Generation APIs

    See fastchat/serve/huggingface_api.py

    OpenAI-compatible RESTful APIs & SDK

    See docs/openai_api.md

    Evaluation

    GPT-4 is the foundation of our AI-enhanced assessment workflow. This section gives a high-level overview of the pipeline. Please see the evaluation documents for further information.

    Pipeline Steps

    1. Create replies using several models: For ChatGPT, use qa_baseline_gpt35.pyor provide the model checkpoint and run get_model_answer.py for Vicuna and other models.
    2. Generate reviews with GPT-4: GPT-4 may be used to generate reviews automatically. If you do not have access to the GPT-4 API, you can do this step manually.
    3. Generate visualization data: Run generate_webpage_data_from_table.py to create data for a static website that will allow you to visualize the evaluation data.
    4. Data visualization: Under the webpage directory, provide a static website. To serve the webpage locally, use python3 -m http.server

    Data Format and Contribution

    For evaluation, they employ a data format encoded with JSON lines. Models, prompts, reviewers, questions, replies, and reviews are all included in the structure.

    By gaining access to the appropriate data, you may customize the evaluation process or contribute to the data.

    For detailed instructions, please refer to the evaluation documentation.

    Also Read ChatGPT4 for Free.

    This article helps you learn about FastChat. We trust that it has been helpful to you. Please feel free to share your thoughts and feedback in the comment section below.

    Share2Tweet2SendShare
    Cloudbooklet

    Cloudbooklet

    Leave a Reply Cancel reply

    Your email address will not be published. Required fields are marked *

    Related Posts

    Chatgpt To Translate

    How to Use ChatGPT to Translate Your Website or Blog

    1 day ago
    Fantasy Minecraft Servers

    5 Best Fantasy Minecraft Servers in 2023

    1 day ago
    Ai Statistics And Trends

    AI Statistics and Trends: What You Need to Know in 2023

    1 day ago
    Block Youtube Ads

    How to Block YouTube Ads on Android TV in 2023 (6 Easy Methods)

    2 days ago

    Follow Us

    Trending Articles

    Microsoft Editor Vs Grammarly

    Microsoft Editor vs Grammarly: Which is the Best Grammar Checker?

    September 18, 2023

    How to Become an AI Trainer: Skills, Salary, and Career Opportunities

    How to Delete Netflix Account Permanently

    Create a Professional Website with Wix AI Website Builder

    5 Best Laptop for Minecraft in 2023: Top Picks for All Budgets

    AI Annotation Jobs: Everything You Need to Know

    Popular Articles

    10 Best Email Tracking App For 2023: Which One Is Right For You?

    10 Best Email Tracking App for 2023: Which One is Right for You?

    September 1, 2023

    How to Find Who Unfollowed You on Instagram

    4 Ways to Reset Snapchat Password

    7 Best AI Content Detectors to Check Plagiarism

    How to Use Hulu Bug Tracker to Improve Your Streaming Experience

    10 Best AI Mockup Generators to Create Stunning Mockups

    Subscribe Now

    loader

    Subscribe to our mailing list to receives daily updates!

    Email Address*

    Name

    Cloudbooklet Logo

    Welcome to our technology blog, where we explore the latest advancements in the field of artificial intelligence (AI) and how they are revolutionizing cloud computing. In this blog, we dive into the powerful capabilities of cloud platforms like Google Cloud Platform (GCP), Amazon Web Services (AWS), and Microsoft Azure, and how they are accelerating the adoption and deployment of AI solutions across various industries. Join us on this exciting journey as we explore the endless possibilities of AI and cloud computing.

    • About
    • Contact
    • Disclaimer
    • Privacy Policy

    Cloudbooklet © 2023 All rights reserved.

    No Result
    View All Result
    • News
    • Artificial Intelligence
    • Applications
    • Linux

    Cloudbooklet © 2023 All rights reserved.