Many .NET applications need to update a database and publish an event when a business operation succeeds. For example, an order service may save a new order in SQL Server and publish an OrderCreated event to Kafka or RabbitMQ so that other services can process the order.
The challenge is that the database and the message broker usually operate as separate systems. A database transaction cannot automatically guarantee that a message will be published successfully to an external broker.
This creates a reliability problem. The order may be saved successfully, but the application can crash before publishing the event. Alternatively, the event may be published, but the database transaction may fail. Either situation can leave the application and its dependent services with inconsistent information.
The transactional outbox pattern addresses this problem by storing the business change and the event to be published in the same database transaction. A separate background worker then reads pending events from the outbox and publishes them to the message broker.
In this article, we will implement the transactional outbox pattern in .NET using Entity Framework Core, understand how the background publisher works, and examine the failure scenarios that matter in production.
What Is the Transactional Outbox Pattern?
The transactional outbox pattern stores messages in a database table as part of the same transaction that updates business data.
Instead of trying to save an order and publish a message as two independent operations, the application performs the following steps:
Begin a database transaction.
Save the order.
Insert an event record into the outbox table.
Commit the transaction.
Let a background worker publish the event.
Mark the outbox record as processed after successful publication.
The key principle is that the business data and the outbox record are committed together.
If the transaction fails, neither change is committed. If the transaction succeeds, the application has durable evidence that the event still needs to be published.
This does not make the database transaction atomic with the message broker. Instead, it removes the dangerous gap between saving business data and recording the intention to publish a message.
Why Direct Message Publishing Can Fail
Consider a typical order creation method:
public async Task CreateOrderAsync(
Order order,
CancellationToken cancellationToken)
{
_dbContext.Orders.Add(order);
await _dbContext.SaveChangesAsync(
cancellationToken);
await _messagePublisher.PublishAsync(
new OrderCreated(order.Id),
cancellationToken);
}This code looks reasonable, but it has a reliability gap.
The database operation may succeed while message publication fails. If the application crashes after SaveChangesAsync completes, the order remains in the database, but no event may be delivered.
Reversing the order of operations does not solve the problem:
await _messagePublisher.PublishAsync(
new OrderCreated(order.Id),
cancellationToken);
_dbContext.Orders.Add(order);
await _dbContext.SaveChangesAsync(
cancellationToken);In this version, the message may be published even if the database transaction fails. Other services may start processing an order that does not exist in the database.
The problem is not simply the order of the two calls. The problem is that they belong to different transactional systems.
How the Outbox Pattern Works
The transactional outbox pattern separates the business transaction from message publication.
Step 1: Business request
The .NET application receives a request to create an order.
Step 2: Single database transaction
The application inserts the order and its corresponding outbox event into the database within the same transaction.
Step 3: Background publisher
A background worker reads pending events from the outbox table and publishes them to the message broker.
Step 4: Message broker
The broker delivers the event to the appropriate consumers.
The database transaction guarantees that the order and the outbox record are committed together. The background publisher handles the separate task of delivering the recorded event.
If the broker is temporarily unavailable, the outbox record remains available for a later retry.
Step 1: Create the Outbox Table
The outbox table stores the event payload and the information required to publish and track it.
The following SQL Server table is a simple starting point:
CREATE TABLE dbo.OutboxMessages
(
Id UNIQUEIDENTIFIER NOT NULL PRIMARY KEY,
Type NVARCHAR(300) NOT NULL,
Payload NVARCHAR(MAX) NOT NULL,
OccurredOnUtc DATETIME2 NOT NULL,
ProcessedOnUtc DATETIME2 NULL,
RetryCount INT NOT NULL DEFAULT 0,
Error NVARCHAR(2000) NULL
);
CREATE INDEX IX_OutboxMessages_Pending
ON dbo.OutboxMessages (OccurredOnUtc)
WHERE ProcessedOnUtc IS NULL;The columns serve different purposes:
Id uniquely identifies an outbox message.
Type identifies the event type.
Payload contains the serialized event.
OccurredOnUtc records when the application created the event.
ProcessedOnUtc records when publication was acknowledged and recorded as complete.
RetryCount tracks publication attempts.
Error stores a brief description of the latest failure.
The filtered index helps locate unprocessed messages without indexing every processed row in the same way.
This schema is illustrative. A production implementation may need fields for the aggregate ID, tenant ID, correlation ID, schema version, lease information, or other operational metadata.
Step 2: Define the Entity Framework Core Entity
Create a C# entity that represents an outbox message.
public sealed class OutboxMessage
{
public Guid Id { get; set; }
public string Type { get; set; } = string.Empty;
public string Payload { get; set; } = string.Empty;
public DateTime OccurredOnUtc { get; set; }
public DateTime? ProcessedOnUtc { get; set; }
public int RetryCount { get; set; }
public string? Error { get; set; }
}Next, register the entity in your application's DbContext.
public sealed class AppDbContext : DbContext
{
public AppDbContext(
DbContextOptions<AppDbContext> options)
: base(options)
{
}
public DbSet<Order> Orders => Set<Order>();
public DbSet<OutboxMessage> OutboxMessages =>
Set<OutboxMessage>();
protected override void OnModelCreating(
ModelBuilder modelBuilder)
{
modelBuilder.Entity<OutboxMessage>(entity =>
{
entity.HasKey(x => x.Id);
entity.Property(x => x.Type)
.HasMaxLength(300)
.IsRequired();
entity.Property(x => x.Payload)
.IsRequired();
entity.Property(x => x.Error)
.HasMaxLength(2000);
entity.HasIndex(x => x.OccurredOnUtc)
.HasFilter("[ProcessedOnUtc] IS NULL");
});
}
}The filter shown here is specific to SQL Server. If you use PostgreSQL or another database provider, configure the filtered or partial index using the syntax supported by that provider.
Create and apply an EF Core migration so the table and index exist in the database.
Step 3: Save the Order and Outbox Event in One Transaction
Now modify the order creation logic so that both the business entity and the event are saved through the same database transaction.
First, define the event:
public sealed record OrderCreated(
Guid OrderId,
decimal TotalAmount);The event contains the information another service needs to react to the new order. Avoid including sensitive customer information unless the receiving system genuinely needs it.
Next, implement the application service:
using System.Text.Json;
using Microsoft.EntityFrameworkCore;
public sealed class OrderService
{
private readonly AppDbContext _dbContext;
public OrderService(AppDbContext dbContext)
{
_dbContext = dbContext;
}
public async Task<Guid> CreateOrderAsync(
decimal totalAmount,
CancellationToken cancellationToken)
{
var order = new Order
{
Id = Guid.NewGuid(),
TotalAmount = totalAmount,
CreatedOnUtc = DateTime.UtcNow
};
var orderCreated = new OrderCreated(
order.Id,
order.TotalAmount);
var outboxMessage = new OutboxMessage
{
Id = Guid.NewGuid(),
Type = nameof(OrderCreated),
Payload = JsonSerializer.Serialize(
orderCreated),
OccurredOnUtc = DateTime.UtcNow
};
await using var transaction =
await _dbContext.Database.BeginTransactionAsync(
cancellationToken);
_dbContext.Orders.Add(order);
_dbContext.OutboxMessages.Add(outboxMessage);
await _dbContext.SaveChangesAsync(
cancellationToken);
await transaction.CommitAsync(
cancellationToken);
return order.Id;
}
}The Order class in this example represents your application's existing order entity.
Both entities are saved within the same transaction. If saving either entity fails, the transaction can be rolled back, preventing the order from being committed without its corresponding outbox message.
If the transaction commits successfully, the outbox event is durable even when the broker is unavailable.
Why Not Publish the Event Inside This Transaction?
A database transaction cannot normally include an ordinary Kafka or RabbitMQ publish operation as part of the same atomic commit.
Publishing while holding the database transaction open also increases transaction duration and may hold locks while the broker is slow or unavailable.
The outbox pattern separates these operations: commit the business state and event record first, then publish the event asynchronously.
Step 4: Define a Message Publisher
The background worker needs an abstraction for publishing messages to your broker.
For example:
public interface IMessagePublisher
{
Task PublishAsync(
string messageId,
string eventType,
string payload,
CancellationToken cancellationToken);
}The implementation can use Kafka, RabbitMQ, Azure Service Bus, or another messaging system.
The messageId is important. Pass the outbox ID through to the broker as a message identifier or header when the chosen broker supports it. Consumers can then use that identifier for deduplication.
The interface does not guarantee delivery by itself. The implementation must await the broker's appropriate acknowledgement and surface publication failures to the caller.
Step 5: Publish Pending Messages in a Background Worker
A hosted service can periodically look for pending outbox records and publish them.
For a simple, single-worker implementation, the following code illustrates the workflow:
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
public sealed class OutboxPublisherService
: BackgroundService
{
private readonly IServiceScopeFactory _scopeFactory;
private readonly IMessagePublisher _publisher;
private readonly ILogger<OutboxPublisherService> _logger;
public OutboxPublisherService(
IServiceScopeFactory scopeFactory,
IMessagePublisher publisher,
ILogger<OutboxPublisherService> logger)
{
_scopeFactory = scopeFactory;
_publisher = publisher;
_logger = logger;
}
protected override async Task ExecuteAsync(
CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
try
{
await PublishPendingMessagesAsync(
stoppingToken);
}
catch (OperationCanceledException)
when (stoppingToken.IsCancellationRequested)
{
break;
}
catch (Exception exception)
{
_logger.LogError(
exception,
"An error occurred while publishing outbox messages.");
}
try
{
await Task.Delay(
TimeSpan.FromSeconds(2),
stoppingToken);
}
catch (OperationCanceledException)
when (stoppingToken.IsCancellationRequested)
{
break;
}
}
}
private async Task PublishPendingMessagesAsync(
CancellationToken cancellationToken)
{
await using var scope =
_scopeFactory.CreateAsyncScope();
var dbContext = scope.ServiceProvider
.GetRequiredService<AppDbContext>();
var messages = await dbContext.OutboxMessages
.Where(x => x.ProcessedOnUtc == null)
.OrderBy(x => x.OccurredOnUtc)
.Take(100)
.ToListAsync(cancellationToken);
foreach (var message in messages)
{
try
{
await _publisher.PublishAsync(
message.Id.ToString(),
message.Type,
message.Payload,
cancellationToken);
message.ProcessedOnUtc = DateTime.UtcNow;
message.Error = null;
}
catch (OperationCanceledException)
when (cancellationToken.IsCancellationRequested)
{
throw;
}
catch (Exception exception)
{
message.RetryCount++;
message.Error = exception.Message;
_logger.LogWarning(
exception,
"Failed to publish outbox message {MessageId}.",
message.Id);
}
}
await dbContext.SaveChangesAsync(
cancellationToken);
}
}Register the worker in your application:
builder.Services.AddHostedService<OutboxPublisherService>();Register AppDbContext and the concrete IMessagePublisher implementation as well.
This example demonstrates the core workflow, but it is not a complete multi-instance production dispatcher. In particular, multiple workers must not independently publish the same pending rows without coordination. The next sections explain the main reliability requirements.
How the Worker Behaves
The worker retrieves a limited batch of unprocessed messages, publishes each message, and marks successfully published messages as processed.
If publication fails, the worker increments the retry count and records an error. Because the message remains unprocessed, a later pass can attempt publication again.
There are several details to consider:
The batch size limits how many messages are loaded at once.
The polling delay controls how frequently the worker checks for new messages.
A new dependency-injection scope provides a scoped
DbContext.Cancellation is propagated to database operations, delays, and message publication.
Failures are logged so operators can investigate persistent problems.
The worker is intentionally simple. It does not yet implement a durable retry schedule, dead-letter handling, distributed locking, or parallel dispatch.
Step 6: Handle the Duplicate-Publishing Window
The outbox pattern prevents the application from losing the intention to publish an event when the database transaction succeeds. It does not guarantee that a message will be published exactly once.
Consider this sequence:
The worker publishes an event to Kafka.
Kafka acknowledges the publication.
The worker crashes before saving
ProcessedOnUtc.The application restarts.
The worker reads the still-pending outbox row and publishes it again.
The same event may be delivered twice.
For this reason, most outbox implementations provide at-least-once publication, meaning messages may be delivered more than once and consumers must handle duplicates safely.
Use an Idempotent Consumer
A consumer can keep a record of message IDs it has already processed.
For example, create an inbox table:
CREATE TABLE dbo.ProcessedMessages
(
MessageId UNIQUEIDENTIFIER NOT NULL PRIMARY KEY,
ProcessedOnUtc DATETIME2 NOT NULL
);The consumer should check and insert the message ID within the same database transaction as its business update.
Conceptually, the process is:
Begin a database transaction.
Insert the message ID into
ProcessedMessages.If the ID already exists, treat the message as a duplicate.
Otherwise, apply the business change.
Commit the database transaction.
Commit the broker offset or acknowledge the message.
If the insert violates the primary key because the message was already processed, the consumer must handle that condition without repeating the business operation.
The inbox record and the business update must share the same transaction. Otherwise, a crash can still leave the deduplication state inconsistent with the business operation.
For external side effects such as sending email or calling a payment provider, database deduplication alone may not be sufficient. Use the external provider's idempotency support where available or apply another durable coordination mechanism.
Step 7: Make the Dispatcher Safe for Multiple Instances
A simple polling worker works as a starting point for one dispatcher. In a scaled deployment, several application instances may query the same pending rows.
Without coordination, two instances can load the same outbox message and both publish it.
A production dispatcher needs a claiming strategy. One option in SQL Server is to claim a batch of pending rows within a short transaction using a lease or claim status, then publish those rows outside that transaction.
A typical design adds fields such as:
LockedUntilUtc
LockedBy
AttemptCount
NextAttemptUtc
The claim transaction selects eligible rows, marks them as claimed, and commits. The worker then publishes the claimed messages and records the outcome.
The lease should expire so another worker can recover messages if an instance crashes. A lease does not eliminate duplicate delivery: a worker may publish successfully, pause, and lose its lease before it records success.
SQL Server implementations often use carefully designed locking and update queries, potentially with UPDLOCK, READPAST, and ROWLOCK hints where appropriate. The exact strategy must be tested against the database's isolation level and workload.
Do not hold a database transaction open while waiting for the broker. Keep claiming and state updates short, and use the message ID for consumer deduplication.
Step 8: Implement Retry and Dead-Letter Handling
Retrying every failure indefinitely can cause problems.
For example, if an event contains an invalid payload, repeatedly publishing it will not fix the underlying issue. The same record may consume resources on every polling cycle.
A production implementation should distinguish transient failures from permanent failures.
Transient failures may include temporary network errors, broker unavailability, or throttling. Permanent failures may include invalid event schemas or unsupported message types.
Use an increasing retry delay, such as exponential backoff with a maximum limit, and consider adding jitter to prevent many workers from retrying simultaneously.
The outbox schema can include a NextAttemptUtc column so the dispatcher only selects messages whose retry time has arrived.
After a configured number of failed attempts, the application can mark a message for investigation or move it to a dead-letter state. Do not silently delete failed messages. Preserve enough information for operators to diagnose and, when appropriate, replay them safely.
A dead-letter policy should include alerts, ownership, retention, and a documented recovery procedure.
Transactional Outbox vs. Direct Publishing
Requirement | Direct publishing | Transactional outbox |
|---|---|---|
Simple implementation | Easier | Requires additional infrastructure |
Business data and event intent committed together | No | Yes |
Recover publication after application crash | Not automatically | Yes, while the outbox record is retained |
Duplicate messages possible | Yes, depending on retries | Yes |
Requires background processing | Not necessarily | Usually |
Supports reliable retry after broker outages | Requires custom recovery | Naturally supports pending-message recovery |
Operational maintenance | Lower initially | Requires monitoring and cleanup |
Direct publishing may be sufficient for noncritical notifications where occasional loss is acceptable. The outbox pattern is more appropriate when downstream services must eventually learn about committed business changes.
Best Practices for a Production Outbox
Store the business change and outbox record in the same database transaction.
Assign every message a stable unique identifier.
Await the broker's acknowledgement before marking a message as processed.
Design consumers to handle duplicate delivery.
Use a claim or lease mechanism when multiple dispatchers can run concurrently.
Implement bounded retries, backoff, and dead-letter handling.
Monitor pending-message count, oldest pending-message age, publication failures, and retry counts.
Define a retention policy for successfully processed outbox records.
Version event payloads so consumers can handle schema changes.
Avoid holding database transactions open while waiting for the broker.
Test crash scenarios, broker outages, database failures, and duplicate delivery.
Keep sensitive information out of event payloads unless it is required and appropriately protected.
Summary
The transactional outbox pattern solves a common reliability problem in distributed .NET applications: saving business data and publishing an event are separate operations, so one can succeed while the other fails.
By writing the business record and its event into the same database transaction, the application ensures that a committed business change has a durable event waiting to be published. A background worker then publishes pending events and records successful delivery.
The pattern does not provide exactly-once delivery across the database and broker. A crash after publication but before marking the outbox record as processed can cause duplicate messages. Idempotent consumers, stable message IDs, and coordinated dispatch are therefore essential parts of a reliable implementation.
For applications that use Kafka, RabbitMQ, or another message broker to coordinate business workflows, the transactional outbox pattern provides a practical foundation for reliable event publication without requiring a distributed transaction across the database and broker.
Join the conversation! Your thoughts help the community grow.