Rebuilding Notion’s lexical search reindexer

Search at Notion has to keep up with every block users create, edit, or delete. Rebuilding our Elasticsearch index used to be a quarterly event that took several weeks. Each rebuild needed constant monitoring and would finish with noticeable gaps.

Today, the same operation finishes in under 24 hours, has 100 percent document consistency, runs end-to-end with zero manual intervention, and can run as often as we need it to.

This is the story of how we got there. We replaced a custom ECS-based indexing system with an Apache Spark–native pipeline, eliminated a second “catchup” pipeline using Elasticsearch’s own primitives, and removed every external dependency from the indexing hot path.

Lexical search refers to the keyword-matching half of Notion’s search: when you type a word into the search bar and want pages containing that word. Every edit in Notion is made searchable by an online indexing pipeline that updates the Elasticsearch cluster. But there are a few reasons we need to occasionally rebuild the entire index from a database snapshot:

• Adding new search features (like making Custom Agents searchable across a workspace); each one requires backfilling the index for every existing block

• Changing how text is analyzed, such as better tokenization for non-Latin scripts

• Migrating to a new Elasticsearch version or launching in a new region

We call this operation “offline reindexing.” It’s what allows us to ship search improvements, and its speed and reliability set a hard ceiling on how fast our Search team could iterate.

Before describing what we built, here’s the shape of the reindexer system we replaced.

Notion’s data lake holds a mirror of every block, a few hours behind real-time. A set of Spark jobs reads that data and produces Apache Avro files. These files are the canonical input to reindexing, and the new system still uses them.

What we replaced was a fleet of ECS Fargate workers running Node. These workers read chunks of Avro data, queried Snowflake for metadata that the Avro files didn’t carry, and wrote to the Elasticsearch cluster using Elasticsearch’s indexing API. It was built in this way to reuse the TypeScript code that turns a Notion block into a searchable document, using the same code that powers online indexing. This gave us perfect parity between the online and offline paths for free.

The downside to this approach, however, is that it required us to run a batch workload on a system designed for online services. Node’s single-threaded event loop meant each worker pinned one core, so scaling out required more containers, not more efficient ones. Per-process memory made every worker re-hydrate the same lookup state. And going through the indexing API put every write on the same path as live traffic, with the locking and version-check overhead that implies.

This first iteration taught us that code reuse alone isn’t a strong enough reason to keep a fundamentally batch workload off a runtime designed for batch processing.

Our first instinct was to throw everything out and start over. We resisted that for two reasons:

1. The document-generation logic, which turns a Notion block into an Elasticsearch document, was the highest-risk code in the pipeline. It encoded years of edge cases around per-document permission resolution, deletion semantics, and field formatting. Rewriting it from scratch in one shot risked introducing silent search-quality regressions.

2. We wanted each step to be independently verifiable, with a small enough blast radius that a regression in one layer wouldn’t cascade. A search index is load-bearing, and debugging many changes at once is very difficult.

We broke the rebuild into four parts. Each step solved a specific pain point and could be deployed and validated before the next one started.

Most of the Snowflake traffic during a reindex wasn’t computing anything immediately necessary, so we removed that lookup from the hot path and replaced it with a nightly Spark job.

The Spark job reads from the data lake and materializes flat files containing everything that the ECS workers need to query from Snowflake. The workers can then read the files from S3 instead of issuing live queries.

This meant that indexing throughput no longer depended on warehouse availability, and we stopped competing with the Data team for compute. It also opened the door for the next step: with no live external dependency, we could move document generation anywhere.

Direct writes to Elasticsearch through the indexing API plateaued around ~200K documents per second, so we changed what we wrote, and where.

Elasticsearch supports a native snapshot format: a self-contained, restore-ready representation of an index on disk, structured as a per-index directory with one subdirectory per shard, a small set of manifest files, and a tree of Lucene segment files. If we could produce those files offline, we could bypass the live-write path entirely and let Elasticsearch’s own snapshot and restore APIs bootstrap the cluster atomically. The same ES-native shape carries through to the final step, where _reindex handles catchup. The live indexing API stays out of the rebuild path entirely.

Producing this format from a Spark job is more subtle than serializing JSON. The shard layout has to match what the live cluster expects, the segment names in the manifests have to match the files on disk, and the per-shard outputs have to add up to something the master node accepts as a coherent snapshot.

The first job, a shard partitioner, repartitions the document stream so each Spark partition corresponds to exactly one Elasticsearch (index, shard) pair. The partitioner wraps the exact routing logic the live cluster uses, both the application-level index assignment (which workspace’s blocks live in which physical index) and Elasticsearch’s own shard ID hash. Documents come out of this stage bit-identically partitioned to where they would land on direct writes.

The second job, a snapshot writer, is where the snapshot files get written. Each Spark task spins up an embedded Elasticsearch node inside the JVM, configured with the index’s real mappings and settings, and issues real IndexRequests against it. The routing key (spaceId), the external-version semantics (version_type: external, sourced from a pipeline docVersion), and the _source handling all match the live pipeline. When the task finishes, the embedded node snapshots itself to a local directory. A Hadoop OutputCommitter then uploads the resulting manifests and Lucene segment files to S3 in parallel. A small manifest tracks each file’s local source path, its destination inside the snapshot repository, and its size, so that commitJob can stitch every task’s output into one coherent snapshot.

We couldn’t pack multiple snapshot tasks onto the same executor. Each Spark task spins up an embedded Elasticsearch node that writes the in-progress snapshot to executor-local disk. The initial assumption was that Spark would happily run several of these per executor, the way it does for any other workload. In practice, two embedded ES tasks sharing one EBS volume would compete for the same local disk and exhaust it. The snapshot bytes for even a single shard are substantial. Two at once on the same executor and we were out of space.

The fix lives in the snapshot writer job, not the partitioner. When it reads each pre-partitioned shard file, .coalesce(1, shuffle = false) collapses the whole file into a single Spark task. Combined with executors sized for one task each (so two embedded ES nodes never share a disk), parallelism ends up across shards, not within them, and the size of the biggest shard sets the wall-clock floor.

Even with one task per executor, EBS was too small and too slow. We started on EBS-backed executors and hit two walls. The first was capacity: some shards were larger than the EBS volume we’d provisioned, and the task ran out of disk partway through. There was no parallelism trick that helped. The second was throughput: embedded ES does a lot of sequential writes during snapshot creation, and EBS sequential-write throughput throttled it. We were paying for embedded ES nodes that spent most of their time waiting on disk.

We moved to instance-store NVMe executors. That made the largest shards completable (on EBS they would simply fail) and gave us meaningfully higher throughput on the shards that did finish. We still always clean up local snapshot files on upload failure. Without that, a single S3 throttle event could exhaust the local disk and cascade into a failed job.

A self-contained snapshot in S3 that Elasticsearch restores in a couple of hours instead of writing for weeks. And because the artifact is just files, atomic deployments and trivial rollback come for free. If a new snapshot regresses, we restore from yesterday.

This step kept the highest-risk piece of code, document generation in TypeScript on ECS, unchanged. The ECS Fargate workers now writes JSONL to S3 instead of to Elasticsearch, and the two Spark jobs pick up from there. The migration to snapshot-based loading was complete before we touched the next layer.

With the snapshot pipeline proven out, the next step was getting rid of ECS entirely. The operational tax was real: workers crashed with OOMs, network errors caused partial writes, and every run started with an engineer hand-computing a memory budget high enough to make progress but low enough that workers would restart themselves before the OS killed them. Each reindex paged on-call five to ten times.

We ported the document-generation logic from TypeScript on Node.js to Scala on Spark. The new job does everything the ECS workers used to do. It reads the Avro output from the Spark prep stage, loads the precomputed cache, resolves per-document permissions, and formats the Elasticsearch document. Instead of writing to ES, it emits JSONL into the same pipeline the shard partitioner already consumed.

Some pieces didn’t translate trivially. Language detection is one example. The TypeScript pipeline used a Node binding for CLD2 (Google’s Compact Language Detector), and the JVM had no equivalent binding. We compiled CLD2 from source as a Linux shared library, packaged it into a tarball, and shipped it to S3 so each Spark task could load it at startup. The Scala job ends up calling into the same underlying C++ library, so language detection output is byte-for-byte identical with the old path, which our field-level validator could verify.

This was the highest-risk step in the migration. The document-generation code is the bridge between Notion’s internal block model and the format Elasticsearch indexes. A subtle behavior change here would show up as a search-quality regression weeks later, when users noticed they could not find their pages.

We leaned heavily on offline validation. Two Spark jobs ran in CI for every change.

An aggregate validator compared the Spark output against the existing TypeScript output at the doc-count and type-distribution level. Are we producing the right number of documents, of the right types?

A field-level validator stratified-sampled documents by block type (so rare types like workflow and form always got coverage) and compared every field, value-by-value. For documents that exist in both pipelines, are the field values identical?

We ran these jobs against sampled production data at every checkpoint. By the time we cut over, we had quantified field-level parity between the two implementations down to a handful of acceptable differences (deterministic ordering of arrays, mostly).

Once we shipped, Spark handled memory management and worker coordination automatically, and Airflow handled orchestration. JVM throughput on Avro processing was meaningfully faster than Node.js, and every engineer at Notion could now debug a reindexing run using the same tools they used for any other Spark job.

By this point, the initial-snapshot side of the pipeline was clean. The other ECS pipeline, which consumed Kafka changes after the snapshot finished to catch up on edits that happened during the multi-day build, was still in place. It still suffered the same problems: OOMs, jest-worker coordination, a JSON-not-Avro code path, and around two days to run.

We replaced it with native Elasticsearch primitives.

The idea is to collect changes as they happen rather than reconcile them afterward.

1. Spin up the new Elasticsearch cluster early and pre-create temporary tmp-* indices.

2. Point write aliases at the temp indices, so that live writes from the online indexing pipeline also land in the new cluster’s temp indices.

3. Build the snapshot in parallel. The Spark pipeline produces snapshot files from the data lake, just as before.

4. Restore the snapshot into the new cluster as snap-* indices, sitting alongside the live tmp-* indices. The restored indices have to be separate because Elasticsearch can’t accept live writes into an index that’s mid-restore. The tmp-* indices stay open for the online indexing pipeline; the snap-* indices come online behind them.

The catchup itself is intentionally simple, but it is not a blind overwrite. Once the snapshot has been restored, we atomically move the write aliases from tmp-* to snap-*. From that point forward, new live writes land in the restored indices. Then we run _reindex from each tmp-* index into its corresponding snap-* index using version_type: external and conflicts: proceed.

That versioning detail is what makes the operation safe and retryable. The restored snap-* indices contain the baseline from the snapshot. The tmp-* indices contain the live-write history that accumulated while the snapshot was being built and restored. Reindexing tmp into snap fills that gap. If a document in tmpis newer than the restored copy, Elasticsearch applies it. If a newer live write has already landed in snapafter the alias swap, _reindex hits a version conflict and leaves the newer document alone.

There is one caveat: _reindex only copies documents into the destination index. It does not automatically reconcile records that moved between the alive and deleted index families. If a block moved from alive to deleted during the catchup window, the newer deleted copy can exist in snap-deleted-* while an older alive copy remains in snap-alive-*. We handle that as a separate phantom-detection step: compare versions across the alive/deleted pair and, when enabled, clean the stale opposite-side copy explicitly.

The key invariant is that freshness comes from document versions, not from assuming the snapshot and reindex windows are perfectly disjoint.

The numbers matter, but the cultural change matters more. Adding a new searchable field used to be a monthlong project. It is now something the search team can ship in days: write the Scala transformation, run the validation jobs, and the next nightly run picks it up. One goal now is to open up that capability to any backend engineer.

The new pipeline unblocks the work we couldn’t ship before. We are already using the new pipeline to land searchable Custom Agents, RTL tokenizers, and other previously-blocked fields, and to heal the long tail of legacy “unmatched” blocks that the old pipeline silently dropped.

Longer term, the snapshot-based architecture is a foundation, not a destination. Time-based indices (so we can write-optimize hot data), federated indexing, and per-language analyzers (now that we know each document’s language) are all on the roadmap.

If working on problems like these sounds interesting, the Search team is hiring.