Portable Data Capsules

A data capsule holds all data of one subject. A subject is a user, a tenant, or an account. The capsule is one signed directory. A browser opens it with no server. The app can import it again with no loss of data.

Use a capsule for GDPR and CCPA access requests, for account moves, and for data portability.

Contents of a capsule

A capsule holds personal data. On Unix, export makes each directory 0700 and each file 0600. Export refuses an output path that is a link.

The viewer has one page for each model. A belongs_to value is a link to the target record. Each record also shows the records that point at it.

Register the models

Add a capsule specification to the GdprRegistry of the app:

use autumn_web::gdpr::{GdprRegistry, ModelRegistration};
use autumn_web::gdpr::portability::CapsuleModel;

let registry = GdprRegistry::new()
    .register(ModelRegistration::hard_delete("users"))
    .register(ModelRegistration::hard_delete("posts"))
    .capsule(
        CapsuleModel::new("users", "id")
            .blob("avatar")
            .exclude("password_hash"),
    )
    .capsule(
        CapsuleModel::new("posts", "author_id")
            .belongs_to("author_id", "users")
            .belongs_to("parent_id", "posts"),
    );

autumn_web::app()
    .state_initializer(move |state| state.insert_extension(registry))
    // ...

• The second argument of CapsuleModel::new is the subject column. Export takes the rows where this column is the subject id.
• primary_key changes the key column. The default is id.
• belongs_to(column, target) adds a link to target.id. Use references(column, target, target_column) for a different key.
• blob(column) marks a column that holds a Blob or a blob key.
• exclude(column) keeps a column out of the capsule.

Every column that a model names must be in the table: the subject column, the key, and each link, blob, and excluded column, and the target column of a link to a model in the same export. A typo fails the export (400), so a capsule never lacks records, blobs, or links. A link cannot target an excluded column either: the capsule would not hold the value that the link points at. The other way round, a custom CapsuleStore must give only the columns that it describes, and a value in the key and subject column, and in each NOT NULL column, of every row: any other row fails the export (400). Export also refuses a column that the store describes twice, or that has a name with anything but ASCII letters, digits, and _ (such as a quoted "display name"): import would refuse the capsule. write_dir refuses such a column too.

Exclude all secrets. Export copies every column that you do not exclude. Exclude password hashes, tokens, and internal flags. Import cannot restore an excluded column, so the column must accept NULL or have a default. An exclusion holds for import too: a capsule made before you excluded a column still holds it, and import refuses that capsule (400) rather than write the column, or a blob it names, back. The same holds when you change the subject column or the key of a model: a capsule made before the change is refused (400), since it no longer holds what the model scopes to one subject.

Check blob ownership. Export copies each blob that a blob column names. If users can write a blob key into a row, make sure that the key belongs to the subject. If a record holds a blob key and no blob store is configured, export fails. A blob that the store does not have is skipped with a warning, but its record still names it: import then fails (400) rather than write a record that points at nothing, or at other bytes under the same key. Import checks the blob columns of the app as it is now too: a capsule made before you marked a column as a blob holds its keys, but not their blobs, so it fails the same way. An excluded blob column is not a blob column of the capsule. If a blob changes while export reads it, export fails with a conflict (409): run it again. A Blob value names its store (provider_id). If it names another store than the configured one (for example after a switch of backend), or no store, export fails with a conflict (409): the same key in the new store can hold other bytes. Only a plain key string names no store.

Set the signing secret

Capsules use [security.signing_secret]. Set it with AUTUMN_SECURITY__SIGNING_SECRET. Export, verify, and import need a secret. Without one, they stop with an error. With the prod profile, the CLI uses the same secret rules as the server: a short secret, or a short key in previous_secrets, stops the command.

Verify also accepts the keys in previous_secrets. Old capsules stay valid after a key rotation.

Use the CLI

autumn data capsule export --subject 42 --out ./capsule-42
autumn data capsule verify ./capsule-42
autumn data capsule import ./capsule-42

The CLI compiles the app. Then it runs the app with the app models, database, blob store, and secret. With -p, the app runs from the directory of that workspace member, so it reads its own autumn.toml and .env.

On a profile that is not dev or test, import needs --force. Add --json to get the raw report. verify uses the signer of an installed CapsuleService, if the app has one. verify runs no migration, and the built-in pools never connect; an app's own with_pool_provider runs as it does for export and import. verify hashes each file as a stream, so a huge file in a capsule from anyone costs no memory.

Use the actuator

The actuator has three POST endpoints. They are available only when actuator.sensitive = true. Install a capsule directory first:

use autumn_web::gdpr::portability::CapsuleDirectory;

.state_initializer(|state| state.insert_extension(CapsuleDirectory::new("var/capsules")))

A capsule name is one plain path segment in the capsule directory. Export makes the name from the time and a count, not from the subject, so a listing of the directory does not show who asked for an export. The endpoints do not follow a link, and do not read or write outside that directory.

Use the Rust API

use autumn_web::gdpr::portability::{
    CapsuleSigner, DataCapsule, PgCapsuleStore, export_subject, import_capsule,
};

let store = PgCapsuleStore::new(pool);
let signer = CapsuleSigner::from_config(&config.security.signing_secret)?;

let capsule = export_subject(registry.capsule_models(), &store, "42").await?;
capsule.write_dir("capsule-42".as_ref(), &signer)?;

let loaded = DataCapsule::read_dir("capsule-42".as_ref(), &signer)?;
import_capsule(&loaded, registry.capsule_models(), &store).await?;

PgCapsuleStore needs the postgres feature. It is not available with sqlite. CapsuleService does the same steps and includes blobs. To use a different CapsuleStore, install a CapsuleService extension. You can do this on SQLite.

Sharded apps must install a CapsuleService. The default store reads the control database, but the rows of a subject are on a shard. With [[database.shards]], the default service is not available (501). Give a CapsuleStore that reads the shard of the subject.

Integrity

verify and import do these checks:

• The signature of manifest.json is correct.
• The SHA-256 of each file agrees with the manifest.
• No file is missing. No file is added. No entry is a link.

Import reads only a capsule that passes these checks. It reads each file one time, so it imports the same bytes that it verified.

On Unix, export and verify open the capsule directory one time and do not follow a link. All reads and writes go through that open directory. If a process changes a directory to a link after the open, the reads and writes do not follow it. Export makes each file new and never replaces a file.

The capsule itself must not be a link. The directories above it are resolved one time, when the capsule is opened. A link among them is followed, as the operator configured it. So only the app user may write to the configured capsule directory and to each directory above it.

Import rules

• Each table in the capsule must be a registered capsule model.
• Import writes parent tables before child tables, by the links of the capsule and of the models as they are now. A cycle of belongs_to links is an error. A link to the same table is permitted.
• Postgres import uses one transaction. If one record fails, import writes no record.
• Import skips generated columns.
• Import moves each serial or identity sequence past the imported keys, in the direction of the sequence. Only the imported keys count, not the rows that the table had before. A sequence makes only start + k * step, so only keys on that path count: with INCREMENT BY 3 from 1, key 500 is not a value it makes and does not move it. A sequence can own a numeric or float column too: an integral key such as "1.00" counts as 1. It does this only after all inserts and deferred constraints succeed, and after it checks every move, so a failed import does not change a sequence.
• A key outside the range of its sequence is a conflict (409). A sequence with CACHE above 1 is refused (501): other sessions can hold cached values that an imported key can take. A CYCLE sequence is refused (501) too: after its last value, it starts again at an imported key.
• The import role needs UPDATE on each sequence it moves, and SELECT to read where the sequence is. USAGE alone is not enough: it cannot tell a sequence restarted with ALTER SEQUENCE ... RESTART WITH from an unused one, and import keeps a pending restart value rather than move the sequence back below it. Without them, import stops before it moves any sequence.
• Import when no other writer uses the tables. Another session can take a value from a sequence and insert its row later. If the capsule holds that key, the import succeeds and the other insert fails with a duplicate key. The same holds for a blob that another writer replaces after import checks it. No lock can prevent this, so run import in a maintenance window.
• Import writes blobs before records. It first checks the bytes of each blob against its manifest entry, that no two entries share a key, and that each target table has the columns of the capsule, with the same types (for a domain, also the same base type) and the same generated status: a capsule made before you dropped, renamed, or retyped a column, or made it generated or ordinary, is refused (400) before any blob is written. A row column that the manifest does not describe, and a row without a value in a column that is NOT NULL in the capsule or in the target table, are refused too, as is a capsule with records for a table that has a NOT NULL column without a default that the capsule does not hold (a column added since, or an excluded one). A domain keeps its base type's modifier, so a domain over numeric(10, 3) and one over numeric(6, 2) are different types. A manifest that names a column twice, or names one with anything but ASCII letters, digits, and _, is refused too. If a blob key holds different bytes or a different MIME type, import stops and writes no blob. If a blob changes while import reads it, or another writer takes a key during the import, import stops with a conflict (409).
• A failed import keeps the blobs that it wrote. Another import of the same capsule can find those bytes, take them as its own, and commit records that point at them, so deleting them could break that import. Run the import again: it finds the same bytes and reuses them. If you give up on the import, delete those blobs yourself; they hold personal data.
• Import needs a blob store with a conditional create (BlobStore::put_if_absent). LocalBlobStore and the S3 backend have one. With a store that has none, import of a capsule with blobs fails (501).
• An imported Blob value names the target store: import sets its provider_id, etag, content_type and byte_size to those of the store that now holds the bytes. A plain key string stays as it is.

Data accuracy

Postgres export reads all models in one snapshot. It reads rows with to_jsonb. Export writes numeric, real, double precision, and money as text, also through a domain. Export writes money and each money[] item as a plain number, so the value does not depend on the lc_monetary locale. The manifest keeps the base type of each domain, so import also reads a domain over money or money[] as plain numbers. Time stamps, bytea, uuid, and arrays keep their exact values. An array with bounds other than 1 (for example [0:2]={1,2,3}) travels as its array literal, so it keeps its bounds.

The subject id must be a valid value of the subject column type. For example, abc for a bigint column is an input error (400). The column type must not change the value: ab123-extra for a varchar(5) column, or 1.234 for a numeric(6, 2) column, is also an input error. The same holds for types that round by themselves: 16777217 for a real column (it reads as 16777216), 1.234 for a money column, or 2024-01-01 12:00 for a date column.

Limits

• Export and import keep the full capsule in memory.
• A record can point at a row of a different subject. Then that row must be in the target database, or the import fails with 409.
• A number in a json or jsonb column can lose digits after the 17th significant digit.
• You cannot import into a different schema.