Many .NET applications need to update a database and publish an event to a message broker as part of the same business operation. For example, an order service may save a new order in SQL Server and publish an OrderCreated event to Kafka so that inventory, payment, and notification services can react to it.
The problem is that a database transaction and a message broker operation are usually separate operations. A database update may succeed while message publishing fails. The reverse can also happen: a message may be published, but the database transaction may roll back.
These failures can leave different services with inconsistent information.
The transactional outbox pattern addresses this problem by storing the business change and the message to be published in the same database transaction. A separate background process reads the stored messages and publishes them to the message broker.
This article explains how to implement the transactional outbox pattern in .NET, how to process pending messages, and what to consider when handling retries and duplicate delivery.
Why Database Transactions and Message Publishing Can Fail
Consider an ASP.NET Core order service that saves an order and publishes an event.
public async Task CreateOrderAsync(
Order order,
CancellationToken cancellationToken)
{
_db.Orders.Add(order);
await _db.SaveChangesAsync(cancellationToken);
await _messagePublisher.PublishAsync(
new OrderCreated(order.Id),
cancellationToken);
}This code looks straightforward, but it contains a consistency problem.
Suppose SaveChangesAsync() succeeds, but Kafka is temporarily unavailable when PublishAsync() runs. The order exists in the database, but downstream services never receive the event.
Another problem can occur if the message is published before the database transaction commits. A consumer might receive an event for an order that is not yet visible in the database. If the database transaction subsequently fails, the event refers to a change that never became permanent.
Using a database transaction around SaveChangesAsync() does not automatically include a separate Kafka publish operation in that transaction.
You could try to coordinate both systems with a distributed transaction, but that introduces additional complexity and requires compatible transaction support. The outbox pattern provides a practical alternative for many event-driven applications.
How the Transactional Outbox Pattern Works
The pattern introduces an outbox table in the same database as the business data.
The application performs two writes in one database transaction:
- Save the business entity, such as an order.
- Save an outbox message describing the change.
If the transaction commits, both records are saved. If it rolls back, neither record is saved.
A background worker then reads unpublished outbox messages and sends them to the message broker.
<text color="secondary" size="sm">The basic flow</text>
<box border radius="lg" padding={3} gap={2}> <box background="surface-secondary" radius="md" padding={3} align="center"> ASP.NET Core order service
<text size="sm" color="secondary">Create order and outbox message</text></box> <box align="center"> <icon name="arrow-down" size="lg" color="secondary" /> </box> <box border radius="md" padding={3} align="center"> One database transaction
<text size="sm" color="secondary">Orders table + Outbox table</text></box> <box align="center"> <icon name="arrow-down" size="lg" color="secondary" /> </box> <box background="surface-secondary" radius="md" padding={3} align="center"> Background outbox worker
<text size="sm" color="secondary">Read pending messages and publish</text></box> <box align="center"> <icon name="arrow-down" size="lg" color="secondary" /> </box> <box border radius="md" padding={3} align="center"> Message broker
<text size="sm" color="secondary">Kafka, RabbitMQ, or another broker</text></box> </box>
The database transaction protects the relationship between the business change and the intent to publish an event. It does not make the database and the broker one atomic transaction. The worker still needs retry logic, and consumers must account for duplicate delivery.
Create an Outbox Table
The outbox table stores messages until the application successfully publishes them.
The following example uses SQL Server:
CREATE TABLE dbo.OutboxMessages
(
Id UNIQUEIDENTIFIER NOT NULL
CONSTRAINT PK_OutboxMessages PRIMARY KEY,
MessageType NVARCHAR(200) NOT NULL,
Payload NVARCHAR(MAX) NOT NULL,
CreatedAtUtc DATETIME2 NOT NULL,
ProcessedAtUtc DATETIME2 NULL,
Attempts INT NOT NULL
CONSTRAINT DF_OutboxMessages_Attempts DEFAULT 0,
LastError NVARCHAR(2000) NULL
);
CREATE INDEX IX_OutboxMessages_Pending
ON dbo.OutboxMessages (CreatedAtUtc, Id)
WHERE ProcessedAtUtc IS NULL;Each row represents a message waiting to be published or a message that has already been processed.
The Id provides a unique identifier that can help detect duplicate delivery. MessageType identifies the event type, and Payload stores the serialized message. CreatedAtUtc records when the message was created.
The ProcessedAtUtc column indicates that the worker has successfully completed its publishing step. Attempts and LastError provide basic retry diagnostics.
For a production implementation, you may also need a lease or lock field, a next-retry timestamp, a dead-letter status, and a concurrency-control mechanism. Those fields become particularly important when multiple worker instances process the same outbox table.
Define the Outbox Entity in .NET
Create a corresponding entity for Entity Framework Core:
public sealed class OutboxMessage
{
public Guid Id { get; set; }
public string MessageType { get; set; } = string.Empty;
public string Payload { get; set; } = string.Empty;
public DateTime CreatedAtUtc { get; set; }
public DateTime? ProcessedAtUtc { get; set; }
public int Attempts { get; set; }
public string? LastError { get; set; }
}Add the entity to your database context:
using Microsoft.EntityFrameworkCore;
public sealed class OrdersDbContext : DbContext
{
public OrdersDbContext(
DbContextOptions<OrdersDbContext> options)
: base(options)
{
}
public DbSet<Order> Orders => Set<Order>();
public DbSet<OutboxMessage> OutboxMessages =>
Set<OutboxMessage>();
}Configure the entity mapping and generate a database migration to create the table and filtered index. Make sure the EF Core model matches the actual SQL schema, including the primary key, column lengths, and nullability.
Save the Order and Outbox Message in One Transaction
Now update the order creation logic so that it saves the order and its event together.
using System.Text.Json;
using Microsoft.EntityFrameworkCore;
public sealed record OrderCreated(
Guid OrderId,
decimal Total);
public sealed class OrderService
{
private readonly OrdersDbContext _db;
public OrderService(OrdersDbContext db)
{
_db = db;
}
public async Task<Guid> CreateOrderAsync(
decimal total,
CancellationToken cancellationToken)
{
await using var transaction =
await _db.Database.BeginTransactionAsync(
cancellationToken);
var order = new Order
{
Id = Guid.NewGuid(),
Total = total
};
var orderCreated = new OrderCreated(
order.Id,
order.Total);
var outboxMessage = new OutboxMessage
{
Id = Guid.NewGuid(),
MessageType = nameof(OrderCreated),
Payload = JsonSerializer.Serialize(orderCreated),
CreatedAtUtc = DateTime.UtcNow,
Attempts = 0
};
_db.Orders.Add(order);
_db.OutboxMessages.Add(outboxMessage);
await _db.SaveChangesAsync(cancellationToken);
await transaction.CommitAsync(cancellationToken);
return order.Id;
}
}The example assumes that the Order entity has Id and Total properties and that the database is configured for Entity Framework Core.
Both entities are saved using the same DbContext and database transaction. If the save or commit fails, the transaction does not leave a successfully committed order without its corresponding outbox message.
The method does not contact the message broker. It only records the business change and the message that needs to be published.
This separation is important. If Kafka is unavailable, the order creation transaction can still succeed, and the outbox worker can retry publishing later.
Build a Background Worker to Publish Messages
A background worker can periodically retrieve pending outbox messages, publish them, and mark them as processed.
The publishing operation should be implemented behind an interface so that the worker does not depend on a specific broker implementation.
public interface IEventPublisher
{
Task PublishAsync(
string messageType,
string payload,
Guid messageId,
CancellationToken cancellationToken);
}The implementation of this interface can publish to Kafka, RabbitMQ, or another supported broker.
A simplified worker might look like this:
using Microsoft.EntityFrameworkCore;
public sealed class OutboxWorker : BackgroundService
{
private readonly IServiceScopeFactory _scopeFactory;
private readonly IEventPublisher _publisher;
private readonly ILogger<OutboxWorker> _logger;
public OutboxWorker(
IServiceScopeFactory scopeFactory,
IEventPublisher publisher,
ILogger<OutboxWorker> logger)
{
_scopeFactory = scopeFactory;
_publisher = publisher;
_logger = logger;
}
protected override async Task ExecuteAsync(
CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
try
{
await ProcessPendingMessagesAsync(
stoppingToken);
}
catch (OperationCanceledException)
when (stoppingToken.IsCancellationRequested)
{
break;
}
catch (Exception ex)
{
_logger.LogError(
ex,
"Error while processing outbox messages.");
}
await Task.Delay(
TimeSpan.FromSeconds(5),
stoppingToken);
}
}
private async Task ProcessPendingMessagesAsync(
CancellationToken cancellationToken)
{
using var scope = _scopeFactory.CreateScope();
var db = scope.ServiceProvider
.GetRequiredService<OrdersDbContext>();
var messages = await db.OutboxMessages
.Where(message => message.ProcessedAtUtc == null)
.OrderBy(message => message.CreatedAtUtc)
.ThenBy(message => message.Id)
.Take(100)
.ToListAsync(cancellationToken);
foreach (var message in messages)
{
try
{
await _publisher.PublishAsync(
message.MessageType,
message.Payload,
message.Id,
cancellationToken);
message.ProcessedAtUtc = DateTime.UtcNow;
message.LastError = null;
}
catch (OperationCanceledException)
when (cancellationToken.IsCancellationRequested)
{
throw;
}
catch (Exception ex)
{
message.Attempts++;
message.LastError = ex.Message;
_logger.LogWarning(
ex,
"Failed to publish outbox message {MessageId}.",
message.Id);
}
}
await db.SaveChangesAsync(cancellationToken);
}
}Register the worker in Program.cs:
builder.Services.AddHostedService<OutboxWorker>();This code demonstrates the basic workflow, but it is not a complete multi-instance production dispatcher. It loads a batch into memory and saves status changes after processing. If multiple worker instances run simultaneously, they may select and publish the same messages.
A production implementation needs safe message claiming, concurrency control, bounded retries, and a clear strategy for messages that repeatedly fail. It should also avoid allowing one poison message to block every later message indefinitely.
Understand the Duplicate Delivery Problem
The outbox pattern prevents the application from losing the intent to publish an event when the database transaction commits. It does not guarantee that each message is delivered exactly once.
Consider what happens when the worker publishes a message successfully but crashes before updating ProcessedAtUtc.
When the worker starts again, the message still appears pending. It may publish the same event a second time.
This is an expected failure mode. The broker may have accepted the event even though the application never recorded the successful result.
For that reason, design the event flow for at-least-once delivery and make consumers idempotent.
A consumer can store processed message IDs in a database table and enforce uniqueness on the ID. The consumer checks and records the message ID in the same transaction as its business update. If the same event arrives again, the consumer can avoid applying the same business operation twice.
For example, a consumer-side table might use a unique key:
CREATE TABLE dbo.ProcessedMessages
(
MessageId UNIQUEIDENTIFIER NOT NULL
CONSTRAINT PK_ProcessedMessages PRIMARY KEY,
ProcessedAtUtc DATETIME2 NOT NULL
);The consumer must still implement the deduplication check and business update atomically. Merely checking the table and then performing a separate update leaves another race condition.
Add Retries and Monitor Failed Messages
Temporary broker outages should not require manual intervention for every failed publish. Retry processing should be part of the outbox design.
Use bounded retries with increasing delays rather than repeatedly retrying a failed message immediately. Store enough diagnostic information to investigate recurring failures, and define what happens when a message reaches its retry limit.
A production outbox may need fields such as:
NextAttemptAtUtcto control retry timing.LockedUntilUtcor a lease token to coordinate workers.- A status field for pending, processing, published, and permanently failed messages.
- A dead-letter destination or administrative recovery workflow.
Do not mark a message as processed until the broker publishing operation has reported success according to the publisher's delivery semantics.
Monitor the number of pending messages, the age of the oldest pending message, publishing failures, retry counts, and processing throughput. A growing backlog can indicate that the broker is unavailable or that the worker cannot publish as quickly as messages are created.
Common Mistakes to Avoid
Publishing before the database transaction commits
A consumer could receive an event for data that was never committed. Write the business data and outbox record together, then publish asynchronously.
Assuming the outbox guarantees exactly-once delivery
The worker can publish a message and fail before recording success. Consumers should be prepared to handle duplicates.
Running multiple workers without coordination
Multiple instances can claim the same pending messages. Use a reliable claiming mechanism, such as row locking or leases, to prevent unnecessary concurrent processing.
Keeping every processed message forever
The outbox can grow indefinitely if completed records are never archived or deleted. Define a retention policy that preserves enough history for troubleshooting and recovery.
Ignoring message ordering
If consumers depend on the order of events for a particular order or customer, the dispatcher must preserve that ordering where required. Parallel publishing can otherwise deliver related events in an unexpected sequence.
Conclusion
The transactional outbox pattern helps .NET applications keep database changes and outgoing events consistent. By saving the business record and its outbox message in the same database transaction, the application avoids losing the intent to publish an event when the message broker is temporarily unavailable.
A background worker publishes pending messages, retries transient failures, and records successful delivery attempts. Because a crash can still cause duplicate publication, consumers should use idempotent processing and reliable deduplication where necessary.
For production systems, pay particular attention to concurrent workers, retries, message ordering, outbox retention, and monitoring. These details turn a basic outbox implementation into a reliable part of an event-driven architecture.
Join the conversation! Your thoughts help the community grow.