Building Reliable Queues in Node.js with Redis and BullMQ

Modern applications perform a great deal of tasks in the background, from sending emails to processing images. These tasks should not slow down user requests, and that is where Node.js background jobs with Redis come in, helping teams build responsive apps. BullMQ makes this setup easier and provides a very simple way to create queues and workers running tasks without blocking the main API. This tutorial will walk you through building a reliable queue system with Redis and BullMQ in a straightforward and easy-to-follow manner.

Queues allow you to run time-consuming work outside of the request cycle, which keeps your app fast, improves the overall user experience, and provides a structured way to scale asynchronous features. You will learn the basics of queues, how BullMQ works, and also how to build a small real project that compresses uploaded images in the background.

Why Modern Apps Need Queues

A queue system makes sure slow tasks do not block the response to the user. A very basic example is an e-commerce platform. When a user uploads a product image, the server must not block to compress the file. Queues allow the API to store this job and return instantly while a worker processes the image in the background. Such a structure keeps performance stable. It also protects your server from overload during high spikes of traffic.

What Redis Does in a Queue System

It acts as the data store for the queue and maintains the details of the job, states, retry counts, and progress. The high speed of Redis makes it ideal for performing background tasks. Redis works quite well with BullMQ, and together, they make a complete scalable task queue architecture for Node.js.

Why BullMQ?

BullMQ is a modern queue library built specifically for Node.js. The BullMQ runs on top of Redis and provides clear APIs for queues, workers, handles retries, schedulers, and rate limiting. BullMQ is also preferable due to predictable performance and simple setup.

You can utilize BullMQ for tasks like email delivery, notification dispatch, media processing, PDF creation, and data ETL jobs.

Install Redis and BullMQ

Let’s start with new project for this example, you can use it into any existing application. Before moving further you need to install redis on your local system or via docker container.

Open your terminal and enter below commands:

npm init -y
npm install bullmq express multer sharp ioredis
touch app.js

It will initiate new project into current directory and app.js file. It will also installed below libraries.

bullmq: A fast and robust queue system for managing background jobs and delayed tasks in Node.js.
express: A minimal and flexible Node.js web framework for building web applications and APIs.
multer: A middleware for handling file uploads, primarily in Express.js applications.
sharp: A high-performance module for server-side image resizing, cropping, and manipulation.
ioredis: A fast and feature-rich Node.js client for connecting and interacting with a Redis database.

Create a Queue in Node.js

For better understanding of queue and workers in Node.js, we will take an example for image processing. In this example, we will take image upload from user via API and compress it using queue job.

queue/imageQueue.js

import { Queue } from 'bullmq'
import { Redis } from 'ioredis'

const connection = new Redis()

export const imageQueue = new Queue('image-compress-queue', { connection })

It will define new queue named as “image-compress-queue”.

API Route to Add Background Job

Let’s modify our app.js file for simple Express API receives an image file and pushes it to the queue. This flow demonstrates asynchronous processing with BullMQ in a clean and easy way

import express from 'express'
import multer from 'multer'
import path from 'path'
import { imageQueue } from './queue/imageQueue.js'

const app = express()
const upload = multer({ dest: 'uploads/' })

app.post('/upload', upload.single('image'), async (req, res) => {
    const filePath = path.join(process.cwd(), req.file.path)
    await imageQueue.add('compress-image', { file: filePath })
    res.json({ message: 'Image received. Compression in progress.' })
})

app.listen(3000)

The queue now stores each job in Redis. The worker will pick them in the background.

Creating a Worker to Process the Job

Now our API and queue is working. Next we need a worker listens to the queue and runs the task. In this case, it compresses the uploaded image using Sharp.

worker/imageWorker.js

import { Worker } from 'bullmq'
import { Redis } from 'ioredis'
import sharp from 'sharp'
import path from 'path'

const connection = new Redis()

new Worker('image-compress-queue', async job => {
    const inputPath = job.data.file
    const outputPath = inputPath.replace('uploads', 'compressed')

    await sharp(inputPath)
        .jpeg({ quality: 60 })
        .toFile(outputPath)

    return { outputPath }
}, { connection })

The worker will compresses each file without blocking your main API. It shows how implementing worker queues in Node.js can improve overall performance while keeping the code simple and clear. Instead of compressing image realtime, we will take image and process it later on queue.

Start the Queue and Test Your Setup

You can run the API and the worker separately so each part handles its own task. This setup is common when working with Node.js background jobs with Redis because it keeps the queue system stable and easy to manage.

Add two commands to your package.json file so you can start everything with a single step.

{
  "scripts": {
    "start:api": "node app.js",
    "start:worker": "node worker/imageWorker.js"
  }
}

Run the API first:

npm run start:api

Open a second terminal and start the worker:

npm run start:worker

Now upload an image through the upload route. The API will accept the file and return instantly. The worker will compress the image in the background and store the new version in the compressed folder. Make sure Redis is already running.

Conclusion

Queuing up background tasks makes systems more reliable, faster, and scalable. BullMQ combined with Redis manages everything behind the scenes to make them a potent duo for Node.js background jobs with Redis. You can use the same setup for many real projects like notification delivery, video processing, report generation, and data syncing. This simple approach will help you build systems that stay fast even under heavy load.

Codewolfy