Typing your Firestore documents with Protobuf

Firestore doesn't have a schema. A document is just a bag of fields: nested maps, arrays, strings, numbers, with no declared shape and nothing stopping two documents in the same collection from looking completely different. For a while that flexibility is convenient. Then the app grows, and the lack of a schema stops being convenient and starts being the thing you fight.

One way out is to give your documents a schema. I was building the backend for a Flutter app and had chosen gRPC, which meant defining a strongly typed API in Protobuf. With protos already describing every message that crosses the wire, aiming the same tool at the Firestore problem seemed like a natural extension. This post walks through that: a second set of proto definitions whose only job is to describe the shape of your Firestore documents, plus a small reflection-based codec to read and write them.

Serializing documents by hand

For the purpose of demonstrating this technique, I'll use snippets from the Java gRPC backend I built for an expense tracking app. One of its core entities is an Expense, stored as a document in an expenses sub-collection under each tracker. The straightforward way to persist it is by hand: each repository gets a converter that turns the domain object into a Map<String, Object> for Firestore, and another that reads the document back. The write side looked roughly like this:

public Map<String, Object> toStorageFormat(Expense expense) {
  Map<String, Object> data = new HashMap<>();

  data.put("id", expense.getId());
  data.put("name", expense.getName());
  data.put("value_micros", expense.getValueMicros());
  data.put("allocation_method", expense.getAllocationMethod().getNumber());

  if (expense.hasCategory()) {
    data.put("category_id", expense.getCategory().getId());
  }

  // Other fields each assembled one by one

  return data;
}

And the read side mirrored it, pulling each field back out by name:

if (doc.contains("name")) {
  builder.setName(doc.getString("name"));
}
if (doc.contains("value_micros")) {
  builder.setValueMicros(doc.getLong("value_micros"));
}
if (doc.contains("allocation_method")) {
  int method = doc.getLong("allocation_method").intValue();
  builder.setAllocationMethod(ExpenseAllocationMethod.forNumber(method));
}

This works. It also means the real schema of an expense document exists only as the intersection of two hand-written methods. Nothing in the codebase says "an expense document has these fields, of these types." To answer that question you read the writer, read the reader, and hope they agree.

They drift. Add a field to the writer, forget the reader, and old documents silently lose data on the next round-trip. Rename a key on one side, miss the other, and you're writing where nobody reads. None of this shows up at compile time, because as far as Java is concerned you're just putting strings into a map. The bug surfaces later, in data, which is the worst place to find it.

There's a deeper problem than drift, and it comes from there being no separate storage type at all: the converter mapped the API Expense straight to a map and back. With nothing in between, the read side quietly grew past decoding into enrichment. paid_by was stored as a user ID, but rebuilt on read into a fully populated User:

if (doc.contains("paid_by")) {
  String paidByUserId = doc.getString("paid_by");
  User paidByUser = participantConverter.buildUser(userManager.getUser(paidByUserId));
  builder.setPaidBy(paidByUser);
}

category_id was worse. Reading it meant walking the document path back to the tracker, fetching that document, reading its owner, and resolving the full category — and the failure path was swallowed.

By then, the converter needed FirebaseUserManager, Firestore, and a category helper just to read a row. With no storage schema to map to, the persistence layer absorbed work — joins, auth resolution, computed fields — that belonged to the API above it.

The pressure scales with the API. As the expense model picks up multi-currency fields, per-participant allocation, and category references, the converters grow in lockstep, and the gap between "what the code writes" and "what you think a document contains" gets wider. What you want is the document shape written down somewhere a compiler can check.

Use different messages for storage and RPCs

Since the gRPC layer already describes every message in a .proto file, the tempting shortcut is to take those API messages as they are and write them straight to Firestore. Protobuf's own style guide warns against exactly that. From the dos and don'ts: Do use different messages for RPC APIs and storage. The reasons it gives are the ones you hit in practice.

The biggest one is computed fields. An API message is shaped by what a client wants to render, so a lot of it is derived at read time and makes no sense to persist. The API Expense is full of these:

• paid_by is a fully resolved User, joined from the stored paid_by_participant_id when the response is built. Storing it would duplicate data that lives elsewhere and go stale the moment that user changes their name.

• allocations is the per-participant breakdown of who owes what, computed from the amounts and the split method. It's the output of a calculation, not input to one.

• amount_in_reporting_currency is derived from the original amount and an exchange rate.

These exist on the wire because the client needs them. None of them belong in the document. Persist the API message and you either store derived data that drifts out of sync, or you litter the message with "ignore this field on write" conventions that no type system enforces.

The split runs the other way too. Storage needs fields the wire should never carry, like the Firebase user ID of whoever created a row, kept for authorization but never exposed. And the two have different invariants: the API rejects states that storage has to tolerate. One allocation config is required-when-present on input, but a background cleanup can legitimately leave it present-but-empty in storage, a document the API validator itself would reject.

So the storage protos become a separate set of messages, in their own package, answering a different question: not "what goes over the wire," but "what does the document look like." The expense API message lives in proto/expense.proto; its storage counterpart lives in proto/storage/expense_storage.proto:

// Represents an Expense document in Firestore
message ExpenseDoc {
  string name = 1;
  expense_tracker.ExpenseAllocationMethod allocation_method = 3;

  // Category reference path. Firestore field type: reference
  optional string category_ref_path = 4;

  int64 created_at_millis = 5;
  int64 updated_at_millis = 6;

  string paid_by_participant_id = 7;
  repeated ExpenseParticipantDoc participants = 8;

  // Original currency, preserved for audit.
  string currency_code = 9;
  int64 amount_micros = 10;

  // Converted to the tracker's base currency, used for all balance math.
  int64 base_currency_amount_micros = 11;
  double exchange_rate_to_base = 12;

  // Firebase user ID of the creator. Used for authorization, never sent on the wire.
  string created_by_user_id = 13;

  // Precomputed allocations, in base-currency micros, keyed by participant ID.
  map<string, int64> participant_allocations = 14;
}

Line this up against the API Expense and the separation pays off. The computed fields are gone. The nested CurrencyAmount of the API is flattened into currency_code, amount_micros, and base_currency_amount_micros, the form the balance calculations actually consume. The category, a full message on the API, is stored as a reference path. created_by_user_id and participant_allocations exist only here. The field numbers don't match the API's either, and they don't need to: these are two independent schemas that happen to describe related things.

Sharing enums between the two

I duplicated the messages but not the enums. ExpenseDoc.allocation_method is the same expense_tracker.ExpenseAllocationMethod the API uses, imported straight from the API proto rather than mirrored into a storage copy.

An enum is a closed set of values with no internal structure to diverge, and the storage values are written by number, not name, so a rename on the wire never touches what's already stored. Mirroring the enum would mean maintaining two identical lists plus a mapping between them for no real gain. The cost I'm accepting is a single shared dependency on a stable, low-churn type.

Serializing protos to Firestore documents

A typed schema is only useful if you're not back to writing it into maps by hand. So the second half is a codec that converts any storage proto to and from a Firestore Map<String, Object> by walking its fields reflectively.

There's a tempting shortcut here that's worth ruling out first. A proto serializes to bytes, and Firestore can store a bytes field, so you could skip the codec entirely and write the serialized message as a single blob. It saves a layer, but it makes the document opaque: Firestore sees one blob, not fields, so you lose indexing, querying by field, partial updates, and anything legible in the console. Mapping to native fields keeps the document a first-class Firestore citizen, which is the reason to be on Firestore rather than a plain key-value store. So the codec targets a field map, not bytes.

Writing is a walk over the fields that are actually set:

public static Map<String, Object> toMap(Message msg) {
  Map<String, Object> result = new HashMap<>();
  for (Map.Entry<Descriptors.FieldDescriptor, Object> entry : msg.getAllFields().entrySet()) {
    Descriptors.FieldDescriptor field = entry.getKey();
    validateNoTimestamp(field);
    result.put(field.getName(), toFirestoreValue(field, entry.getValue()));
  }
  return result;
}

getAllFields() returns only the fields that are present, so unset fields stay out of the document instead of being written as zero values. The proto field name (snake_case) becomes the Firestore key directly, which keeps documents readable and the mapping trivial. toFirestoreValue handles the type translation: nested messages recurse through toMap, maps and repeated fields convert element by element, enums are written as their integer number.

Reading reverses it, and is deliberately forgiving:

public static <T extends Message> T fromMap(T defaultInstance, Map<String, Object> data) {
  Message.Builder builder = defaultInstance.newBuilderForType();
  for (Map.Entry<String, Object> entry : data.entrySet()) {
    Descriptors.FieldDescriptor field =
        defaultInstance.getDescriptorForType().findFieldByName(entry.getKey());
    if (field == null) {
      continue; // a key in the document with no field in the schema: ignore it
    }
    Object value = fromFirestoreValue(field, entry.getValue());
    if (value != null) {
      builder.setField(field, value);
    }
  }
  return (T) builder.build();
}

Both are static helpers, invoked with the storage type's default instance — ProtoFirestoreCodec.fromMap(ExpenseDoc.getDefaultInstance(), data) — so callers get back a typed ExpenseDoc, never a raw Message. That default instance carries the descriptor the reflective walk needs, and it's the only argument the codec takes beyond the data itself.

The layering that results has two translators, each with one job:

A mapper translates between the API message and the storage message, where the deliberate field differences get reconciled. The codec translates between the storage message and the Firestore document, mechanically. The hand-written per-field converters are gone, replaced by one reflection pass that every storage type shares.

Pairing the document with its ID

One field is absent from ExpenseDoc: the id. A document's ID is its key in the collection, not part of its body, and Firestore returns it separately from the field data on every read. Storing it inside the document too would just duplicate the key, with the usual risk of the copy drifting from the real thing.

The code above the codec still needs the two together, though — a stored expense isn't useful without knowing which document it came from. So each storage type gets a small record that pairs the ID with its typed doc:

public record ExpenseRecord(String id, ExpenseDoc doc) {}

The codec produces the doc; the repository reads the ID off the DocumentSnapshot (new ExpenseRecord(snapshot.getId(), doc)) and hands the pair upward. It's a one-line class per collection, and it keeps the key-versus-body distinction explicit instead of smuggling the ID into the schema where it doesn't belong.

One repository per collection

The codec and the records come together one layer up, in a repository class per domain object — ExpenseStorageRepository, ContactStorageRepository, and so on. Each owns a single collection: it holds the path, runs the codec on every read and write, and returns the typed *Record. The raw document representation stays sealed inside — a DocumentSnapshot on the way out, a Map<String, Object> on the way in, never cross that boundary. The service layer above works only with storage protos and records, so the choice of Firestore as the store stays an implementation detail of one class per collection, which is also what makes the repositories straightforward to stub in a test.

What sits between the wire and Firestore

The record and the repository are two pieces of a fixed skeleton that every domain object repeats. Seeing the whole stack for one object is the fastest way to adopt the pattern, so here's the lineup for Expense, from the wire down:

• ExpenseServiceImpl — the generated gRPC entry point. Thin on purpose: it unwraps the request, hands off, writes the response back to the stream. No logic lives here.

• ExpenseOperationsService — the orchestrator, and the only class that knows the whole operation. It authorizes the caller, runs validation in order, reads whatever related collections it needs, and performs the write.

• ExpenseValidator — rejects bad input. It works on the API message, before anything is mapped or stored, which is why the storage layer below it can assume the data is already well-formed.

• ExpenseMapper — translates API Expense to ExpenseDoc and back. The deliberate field differences from earlier — dropped computed fields, the flattened currency, the storage-only created_by_user_id — are reconciled here and nowhere else.

• ExpenseStorageRepository — owns the expenses collection, runs the codec on every read and write, returns ExpenseRecord.

• ProtoFirestoreCodec — the one shared reflective codec; the only class in the list not named after the domain object, because it doesn't need to be.

A write walks straight down that column. Stripped of the transaction wrapper and the balance recomputation, which aren't the point here, createExpense is just the layers called in order:

public Expense createExpense(CreateExpenseRequest request) throws ... {
  String trackerId = request.getExpenseTrackerId();
  authz.requireCreateExpenseAccess(trackerId);                    // 1. authorize
  Expense expense = request.getExpense();

  expenseValidator.validateCreateRequest(request, trackerDoc, count);  // 2. validate the API message
  expenseValidator.validateParticipantsExist(expense, validParticipantIds);
  expenseValidator.validateCategoryExists(expense, validCategoryIds);

  ExpenseDoc doc =                                                // 3. map API -> storage proto
      expenseMapper.toStorageDocForCreate(expense, accountId, baseCurrency);
  ExpenseRecord saved =                                           // 4. codec writes the doc, returns id + doc
      expenseStorageRepository.createInTransaction(txn, trackerId, doc);

  return expenseMapper.toApiExpense(saved, ...);                  // 5. map back, re-deriving the computed fields
}

The read path runs the same column in reverse: the repository's codec turns the document into an ExpenseDoc, and toApiExpense maps it up to the wire message — re-deriving the OUTPUT_ONLY fields (paid_by, allocations, amount_in_reporting_currency) that the storage proto deliberately doesn't hold. The computed fields the storage schema dropped are exactly the ones the mapper fills back in on the way out.

Every collection repeats this column. Adding one is mechanical: a storage proto, a *Record, a *StorageRepository, a *Mapper, and the validator and service that sit above them. The codec is the single piece they all share.

Why this matters more as the app grows

The payoff that's easy to underrate going in is documentation. The shape of every Firestore document is now a typed .proto you can open and read in one place. Without it, the only way to know what a document contains is to read its serializer, read its deserializer, and reconstruct the shape in your head, for every collection. That's manageable for the first few collections; it stops scaling as they multiply and the API above keeps moving. You can't safely change a storage layer you can't see.

The same property pays off if you build with AI coding agents. An agent's working memory is its context window, and a declarative schema is a dense way to fill it. A few things make agentic changes to the storage layer reliable:

• Keep the shape declarative and in one file. An agent changing how a document is stored does far better loading one .proto than inferring the shape from read and write code spread across several classes. Inference is where it drifts, the same way a human does, and it reintroduces exactly the read/write mismatch the pattern set out to kill.

• Let the generated types do the checking. A mismatched field fails to compile, where stringly-typed map code would have silently written bad data. That compile error is feedback the agent can act on in its own loop, without you refereeing.

• Keep the change shape small and repeatable. "Add a field" becomes: edit the proto, regenerate, adjust the mapper. An agent executes that reliably, and a review verifies it at a glance.

It also helps to write the convention down where the agent will read it. In this project the storage-proto pattern and the field-adding steps live in the repo's agent instructions (a CLAUDE.md or AGENTS.md file), so a fresh agent follows them by default instead of improvising a one-off serializer:

### Firestore Field Naming

Proto field names (snake_case) map directly to Firestore document keys.
When adding a new field:

1. Define it in the storage proto (proto/storage/*.proto)
2. No manual mapping needed — ProtoFirestoreCodec handles it
3. Enum values are stored as integers
4. Use int64 for timestamps (milliseconds since epoch)

Steps 3 and 4 are the same rules the codec enforces, written back as guidance. The "no manual mapping" in step 2 is about the storage-to-Firestore hop, which the codec handles on its own; the API-to-storage mapper still needs a line when the new field is also exposed on the wire.

If your Firestore documents have outgrown the point where you can hold their shape in your head, give them a real schema. Protobuf is a good language to write that schema in, doubly so if your API already speaks it and an agent is doing most of the typing.