How to Deploy CogVLM to AWS EC2

First, we need to create an AWS EC2 instance. EC2 is Amazon’s compute product that you can use to deploy virtual machines. Search for “EC2” in the search bar and navigate to EC2.

‍

On the EC2 homepage, click the “Launch instance” button:

‍

This button will take you to a page where you can configure the machine to create.

We recommend choosing the Amazon Linux operating system, which has been optimized for use in AWS. You will need to run Inference on a GPU device to run CogVLM. We recommend choosing a machine image optimized for deep learning, such as the Deep Learning Base GPU AMI image. This image will come with some tooling out of the box that will minimize GPU setup. If you want to deploy on a CPU device, the standard Amazon Linux operating system is recommended.

‍

Once you have configured the virtual machine, you can deploy the system.

Next, sign in to your server with SSH. Read the AWS EC2 SSH instructions to learn more.

‍

Create a new Python file and add the following code:

import base64
import os
from PIL import Image
import requests

PORT = 9001
API_KEY = ""
IMAGE_PATH = "forklift.png"


def encode_base64(image_path):
    with open(image_path, "rb") as image:
        x = image.read()
        image_string = base64.b64encode(x)

    return image_string.decode("ascii")

prompt = "Read the text in this image."

infer_payload = {
    "image": {
        "type": "base64",
        "value": encode_base64(IMAGE_PATH),
    },
    "api_key": API_KEY,
    "prompt": prompt,
}

results = requests.post(
    f"http://localhost:{PORT}/llm/cogvlm",
    json=infer_payload,
)

print(results.json())

This code will make a HTTP request to the /llm/cogvlm route on your Inference installation. This route accepts text and images which will be sent to CogVLM for processing. This route returns a JSON object with the text response from the model.

Above, replace:

1. ROBOFLOW_API_KEY with your Roboflow API key. Learn how to retrieve your Roboflow API key.

3. image.png with the image that you want to use to make a request.

4. prompt with the question you want to ask.

Let’s run the code on the following image of a forklift and ask the question “Is there a forklift close to a conveyor belt?”:

Here is an example output from the script above:

{
    'response': 'yes, there is a forklift close to a conveyor belt, and it appears to be transporting a stack of items onto it.',
    'time': 12.89864671198302
}

‍