Estimating fps with different GPU's

[Snitte]

So im looking to buy a workstation to run a program based on YOLOv5s
im looking to get 120fps at 1080p from my camera, will a p4000 gpu be able to manage that?

[glenn-jocher]

@Snitte I don't have much experience with the wider range of enterprise GPUs NVIDIA seems to be throwing at the market these days. You can benchmark the most common enterprise ones in a few minutes with a GCP or AWS instance (i.e. T4, K80, P100, V100, A100), and you can benchmark the consumer GPUs at www.vast.ai similarly (by pulling the ultralytics/yolov5 container from the container options). In general a 3090 is almost as fast as a V100.

Lambda labs has a few good benchmarks here for comparison shopping. https://lambdalabs.com/gpu-benchmarks

[glenn-jocher]

@Snitte BTW you should also note the benchmarks in https://github.com/ultralytics/yolov5#pretrained-checkpoints are based on --batch-size 32 throughput. Smaller batch sizes will be slower per image, maybe half as fast at batch size 1.

[Snitte]

Hey thanks for the quick reply, would batch sizes not only be relevant during training of a model, not the detection?

[glenn-jocher]

Inference speeds are always a function of batch size, regardless of whether the inference is executed during training or otherwise. In any case this is trivial to test:

import torch
from PIL import Image


# Model
device = torch.device('cuda:0')
model = torch.hub.load('ultralytics/yolov5', 'yolov5s', pretrained=True).to(device)

# Images
for f in ['zidane.jpg', 'bus.jpg']:  # download 2 images
    print(f'Downloading {f}...')
    torch.hub.download_url_to_file('https://github.com/ultralytics/yolov5/releases/download/v1.0/' + f, f)
imgs = [Image.open('bus.jpg')]  # batched list of images

# Inference
%timeit [model(imgs) for _ in range(32)]  # batch-size 1 
%timeit model(imgs * 32)  # batch-size 32