How Infoveave Uses Background Jobs for Data Pipelines
Scheduled ingestion, quality checks, and data automation workflows in Infoveave run as persistent background jobs — not inline HTTP handlers. Hangfire-backed workers process connector syncs, transformation batches, and alert fan-out with idempotency keys so retries never duplicate writes. When you design jobs for a unified data platform, the same rules apply: persist before execute, pass cancellation tokens, and treat every job as retryable.
Definition
Background jobs are tasks that run outside the HTTP request cycle — scheduled syncs, report generation, and queue consumers — so user-facing APIs stay fast and work survives process restarts.
What is a Background Job?
Work that runs outside the HTTP request/response cycle.
Instead of making the user wait — hand the task off and return immediately.
Real-world examples:
Sending a welcome email after sign-up
Generating a PDF report
Resizing an uploaded image
Syncing data with a third-party API
Cleaning up old database records every night
Rule of thumb: If it takes more than ~200ms or doesn't affect the response — move it to the background.
The Four Approaches (from simple to powerful)
Pick the simplest tool that meets your needs.
1. BackgroundService — The Built-in Way
The foundation of background work in .NET. Runs alongside your web app, starts with it, stops with it.
Best for: Polling loops, periodic tasks, simple timers.
csharp
public class CleanupService : BackgroundService
{
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
var timer = new PeriodicTimer(TimeSpan.FromHours(1));
while (await timer.WaitForNextTickAsync(stoppingToken))
{
await DeleteExpiredSessionsAsync();
}
}
}
// In your API — enqueue the work
await _channelWriter.WriteAsync(new EmailMessage(user.Email));
// In your BackgroundService — process the work
await foreach (var msg in _channelReader.ReadAllAsync(ct))
await SendEmailAsync(msg);
Limitation: Still in-memory. App restart = lost queue.
When you need complex cron scheduling, misfire handling, or jobs running across multiple servers.
Best for: Enterprise scheduling, clustered environments.
csharp
// Run at 9am on the 1st of every month
q.AddTrigger(opts => opts
.ForJob(jobKey)
.WithCronSchedule("0 0 9 1 * ?"));
Common cron expressions:
Expression
Meaning
0 * * * * ?
Every minute
0 0 9 * * MON-FRI
9am, Monday to Friday
0 0 0 ? * SUN
Midnight every Sunday
Use Quartz when Hangfire's scheduling isn't enough. For most apps, Hangfire is simpler.
5. Cloud-Native Options
When your system is distributed across multiple services or teams.
Option
Platform
Best For
Azure Service Bus
Azure
Microservices, reliable messaging
Azure Functions
Azure
Serverless, scale-to-zero
AWS SQS + Lambda
AWS
AWS-native architectures
Redis Streams
Any
Ultra-low latency queuing
These shine when jobs need to cross service boundaries or when different teams own different parts of the system.
Choosing the Right Tool
Do jobs need to survive a restart?
│
├── No ──→ BackgroundService or Channels
│
└── Yes ──→ Need complex scheduling?
│
├── Yes ──→ Quartz.NET
│
└── No ──→ Running on Azure / AWS?
│
├── Yes ──→ Service Bus / SQS
│
└── No ──→ Hangfire
For most .NET web apps: Hangfire is the right answer.
3 Rules for Production-Ready Background Jobs
1. Always Handle Cancellation
The app needs to shut down gracefully. Respect the CancellationToken everywhere.
csharp
// Pass the token to every async call
await DoWorkAsync(stoppingToken);
2. Always Design for Idempotency
Jobs can be retried. Running the same job twice should be safe.
csharp
// Check before acting
if (user.WelcomeEmailSentAt.HasValue) return; // already done
await SendEmailAsync(user);
user.WelcomeEmailSentAt = DateTimeOffset.UtcNow;
await db.SaveChangesAsync();
3. Always Log with Context
You can't attach a debugger to a background job. Logging is your only window.
csharp
_logger.LogInformation("Processing job {JobId} for user {UserId}", jobId, userId);
Common Mistakes to Avoid
Mistake
Why It's a Problem
Fix
Using HttpContext in a background thread
It's disposed after the response
Copy values before going async
Calling .Result or .Wait()
Blocks threads, causes starvation
Always use await
Not handling exceptions in loops
Silently kills the service
Wrap in try/catch, log errors
Fire-and-forget without error handling
Exceptions are swallowed
Use Hangfire or log in a try/catch
Jobs that aren't idempotent
Retry = duplicate work or corruption
Add a "processed" guard
Summary
BackgroundService
Channels
Hangfire
Quartz.NET
Cloud
Built into .NET
Yes
Yes
No
No
No
Survives restarts
No
No
Yes
Yes
Yes
Dashboard
No
No
Yes
No
Yes
Cron scheduling
Basic
No
Yes
Yes
Yes
External dependency
None
None
Database
Optional DB
Cloud
Complexity
Low
Low
Medium
Medium
High
Start simple. Add complexity only when you need it.
Most apps need nothing more than Hangfire + PostgreSQL.
Infoveave's data automation layer is built around exactly these patterns. Scheduled data ingestion pipelines, event-triggered transformations, and nightly reconciliation jobs all run as background processes — separate from the API request cycle so the platform stays responsive regardless of how much data is moving. Every pipeline job is designed to be idempotent: if a run fails mid-way and retries, it produces the same result without duplicating records. If you work with data pipelines at scale, the same rules apply whether your job is sending an email or syncing ten million rows from an ERP.
This article was written by the Infoveave Engineering Team — building Unified Data Platform, agentic BI, and enterprise analytics infrastructure. Infoveave (by Noesys Software) helps organisations unify data, automate business processes, and act faster with AI-powered insights.