Integrate Background Jobs & Queue System

The route

6 steps to Done

1. 01

   Add queue library dependency

   Install a production‑ready job library and its broker client.

   Preview prompt + verify gate ▾ Hide ▴

   Edit the project's package.json to include the latest stable versions of "bullmq" and "ioredis" under dependencies. Then run the appropriate install command (npm install or yarn). Ensure the new entries are correctly formatted JSON and that the lockfile is updated. Commit the changes to version control if applicable.

   • ✓ Do bullmq and ioredis appear in package.json with exact version strings?
   • ✓ Does node_modules contain directories for bullmq and ioredis?
2. 02

   Configure Redis connection

   Create a reusable Redis client that the queue will use.

   Preview prompt + verify gate ▾ Hide ▴

   In the src/ folder, create a file named redis.js. Import IORedis from "ioredis". Initialize a new IORedis client using process.env.REDIS_URL if defined, otherwise default to "redis://127.0.0.1:6379". Export the client as the default export. Add error handling: log any connection errors to console. Ensure the file uses CommonJS or ES modules consistent with the existing codebase. Do NOT include mock objects; the client must be a real IORedis instance.

   • ✓ Does src/redis.js exist and export a client?
   • ✓ Does a node test script show "PONG" from client.ping()?
3. 03

   Create a base job queue

   Set up a BullMQ Queue instance that other jobs can use.

   Preview prompt + verify gate ▾ Hide ▴

   Add a file src/queue.js. Import { Queue } from "bullmq" and the Redis client from ./redis.js. Instantiate a new Queue called "appQueue" with the Redis connection (pass the client.connection). Export the Queue instance. Set default job options: attempts: 3, backoff: { type: "exponential", delay: 5000 }. Do NOT comment out the export or use placeholder code.

   • ✓ Does src/queue.js exist and export a Queue instance?
   • ✓ Does a runtime check print "appQueue" among registered queues?
4. 04

   Implement a sample background job

   Demonstrate real work performed asynchronously.

   Preview prompt + verify gate ▾ Hide ▴

   Create src/jobs/fetchUrlJob.js. Import { Worker } from "bullmq", the Queue from ../queue.js, and the DB client (assume an existing Prisma client exported from src/db.js). Define a processing function that receives a job with data { url: string }. Use node-fetch (add as dependency if missing) to GET the URL, capture the response body as text, and insert a record into the "job_results" table with fields: url, status (200 or error code), content (truncated to 5000 chars). Handle errors: on catch, insert status = error.code or 500 and content = error.message. Export a function enqueueFetchUrl(url) that adds a job to appQueue with { url }. Also create a Worker named "fetchUrlWorker" attached to appQueue that runs the processing function. Ensure the worker runs continuously (no placeholders). Do NOT use console.log placeholders for DB actions.

   • ✓ Does enqueueFetchUrl add a job to appQueue (check queue length increase)?
   • ✓ After a job runs, is there a new row in job_results with correct url and status?
5. 05

   Integrate job enqueuing into existing app flow

   Trigger background processing from a user action (e.g., POST /api/urls).

   Preview prompt + verify gate ▾ Hide ▴

   In the existing Express router file (e.g., src/routes/api.js), import { enqueueFetchUrl } from ../jobs/fetchUrlJob.js. Add a POST endpoint at "/urls" that expects JSON body { url: string }. Validate that url is a non‑empty string; if invalid, respond 400. If valid, call enqueueFetchUrl(url) and respond 202 with { message: "Job queued" }. Do NOT return placeholder data or simulate the enqueue; ensure the call reaches the real BullMQ queue. Add error handling to catch any queue errors and return 500 with an error message.

   • ✓ Does POST /api/urls return 202 and a jobId?
   • ✓ Is a new job visible in appQueue after the request?
6. 06

   Run the worker process and verify end‑to‑end execution

   Start the background worker and confirm a full cycle from request to DB persists.

   Preview prompt + verify gate ▾ Hide ▴

   Create a new file src/worker.js. In this file, import the fetchUrlWorker side‑effect from ../jobs/fetchUrlJob.js (the file should export the Worker instance). Ensure the script keeps the process alive (no immediate exit). In package.json, add a script entry: "worker": "node src/worker.js". Install any missing dependencies (e.g., node-fetch, prisma). Verify that running npm run worker starts the BullMQ worker without exiting, logs a startup message, and begins processing queued jobs. Do NOT include placeholder setTimeouts to keep the process alive; rely on the Worker event loop.

   • ✓ Does "npm run worker" keep the process alive and log startup message?
   • ✓ After posting a URL, is a matching row created in job_results?