38. Transactional Outbox & Durable Effects

Use a transactional outbox when a committed database change must eventually produce an external effect even if the application process stops between the two operations. Typical examples are a webhook, an e-mail request, a search projection or a message published to another service.

after_commit is the simpler choice for a best-effort, process-local effect. Outbox is the durable choice. It stores an event in the same relational transaction as the domain mutation, then lets an independent worker claim and deliver it later.

1. Register the schema as a migration

Applications using the built-in SQLx migration runner can register the versioned migration supplied by the ORM:

#![allow(unused)]
fn main() {
use rullst_orm::OutboxMigration;
use rullst_orm::schema::migration::Migration;

fn migrations() -> Vec<Box<dyn Migration>> {
    vec![Box::new(OutboxMigration)]
}
}

In a generated project, add Box::new(rullst::db::OutboxMigration) to the vector returned by get_migrations() in src/migrations/mod.rs, then run cargo rullst db:migrate.

Outbox::install() applies the same idempotent DDL directly and is useful in tests or explicit local setup. It never runs automatically. A production team should review and track the migration through the same deployment process as its domain schema; do not create tables opportunistically while serving a request.

The built-in contract supports the SQLx relational backends: SQLite, PostgreSQL, MySQL and MariaDB. It does not span MongoDB, DuckDB, Turso or SurrealDB transactions.

2. Commit domain state and event together

Choose a stable stream boundary and a deterministic event key. A stream can represent an application or tenant, but it is only a namespace: the framework does not infer authorization from it.

#![allow(unused)]
fn main() {
use rullst_orm::{Error, Orm, Outbox};
use serde_json::json;

async fn create_invoice() -> Result<(), Error> {
Orm::transaction(|_| Box::pin(async move {
    // `?` placeholders are SQLite/MySQL syntax; PostgreSQL uses `$1, $2`.
    let insert = sqlx::query(
        "INSERT INTO invoices (id, status) VALUES (?, ?)"
    )
    .bind(123_i64)
    .bind("issued");
    rullst_orm::execute_query!(insert, execute, pool)?;

    Outbox::enqueue(
        "tenant-42",
        "invoice:123:issued:v1",
        "invoice.issued",
        &json!({"invoice_id": 123}),
    ).await?;

    Ok::<(), Error>(())
})).await?;
Ok(())
}
}

If the transaction rolls back, neither row survives. This also holds when a helper wraps enqueue in its own Orm::transaction and is called from inside another one: the nested call joins the outer transaction through a savepoint, so its outbox row commits or rolls back with the outer transaction. Calling enqueue outside Orm::transaction fails instead of silently opening an unrelated transaction. When the application already owns a raw SQLx transaction, use enqueue_with_tx(&mut transaction, ...).

(stream, event_key) is unique. Repeating the same kind and an equal JSON payload returns the existing event ID with inserted == false, also when a concurrent transaction committed that key after your transaction’s first read (MySQL/MariaDB read the row back with a locking read). Payloads are compared as JSON values, so an object serialized with its keys in another order (for example one built from a HashMap) is the same content; array order and every value still count. Reusing that key for different content is an error; it does not overwrite the original event, and the error does not echo the stream or key.

Streams and event keys are case-sensitive on every backend: order:aB3x and order:Ab3X are different keys, and a worker for tenant-a does not claim events of tenant-A. MySQL/MariaDB declare the key columns with the binary ascii_bin collation for this. A MySQL/MariaDB table created by an earlier version used the server’s case-insensitive default collation; run Outbox::install() once, for example from a new migration, to convert it in place. The conversion rebuilds the table, so schedule it like other DDL.

3. Claim, deliver and acknowledge

Each worker supplies a stable identifier, lease duration and maximum number of claims:

#![allow(unused)]
fn main() {
use rullst_orm::{Error, Outbox};

async fn deliver() -> Result<(), Error> {
if let Some(event) = Outbox::claim_next(
    "tenant-42",
    "webhook-worker-1",
    30, // lease seconds
    8,  // maximum claims
).await? {
    let payload = event.payload()?;

    // Perform the external effect. Pass event.event_key to a provider that
    // supports idempotency, or deduplicate it in the receiving service.
    let delivered = payload.get("invoice_id").is_some();

    if delivered {
        let owned = Outbox::acknowledge(event.id, event.claim_key).await?;
        if !owned {
            // The lease expired or another worker reclaimed the event.
            // Do not assume ownership or mutate the newer claim.
        }
    } else {
        Outbox::fail(
            event.id,
            event.claim_key,
            "provider temporarily unavailable",
            8,  // maximum attempts
            15, // retry delay seconds
        ).await?;
    }
}
Ok(())
}
}

An ACK or failure transition succeeds only for the exact, unexpired random claim token. A failed delivery becomes pending after the bounded delay, or dead_letter when its attempt limit is reached. If a worker dies during its final claim, the next claim sweep moves that expired event to dead-letter instead of retrying forever.

Claims, acknowledgements and failures commit on their own. Calling them inside Orm::transaction returns a validation error, because an acknowledgement there would already be durable if the handler’s transaction later rolled back. Commit the handler’s database work first, then acknowledge.

4. Understand the guarantee

The delivery guarantee is at least once:

the worker claims an event;
the external provider accepts the effect;
the process stops before the database ACK;
the lease expires and the event is delivered again.

No local database design can atomically commit an arbitrary remote HTTP side effect. Make the consumer idempotent using the stable stream/event key. Do not use a random key on every retry.

Other explicit limits:

key fields are 1–128 characters from a bounded ASCII grammar;
JSON payloads are at most one MiB;
leases are 1–3,600 seconds, attempts are 1–100 and retry delays are 0–86,400 seconds;
a failure message is 1–512 bytes without control characters;
ordering is not guaranteed across retries or concurrent workers;
generated observers are not automatically converted into outbox events;
cleanup, retention, tenant authorization and the worker supervision loop belong to the application;
monitor retry and dead-letter state through your normal database operations.

This separation keeps model saves predictable while giving applications a durable primitive where the event schema and operational policy are explicit.

5. Relay into `rullst-messaging`

With the umbrella messaging-orm-outbox feature, OrmOutboxRelay<B> maps one exact outbox stream to one topic on any static MessageBroker. It validates the claimed JSON, uses the stream-scoped event_key (the lowercase hex SHA-256 of the stream, /, then the event key) as the broker idempotency key, publishes and then acknowledges the exact ORM claim. Its executable crash-window test stops after the first publish, reclaims the expired event and observes an exact broker replay with only one retained message.

That bridge improves composition; it does not change the guarantee. ORM commit, broker publish and ORM ACK are not one distributed atomic transaction. Operate the claim loop, retry/dead-letter policy, cleanup, authorization and final consumer deduplication explicitly. See Bounded Brokered Messaging for the relay code.

Keyboard shortcuts