Dropbox has announced the open-sourcing of "Witchcraft," a new local search engine built entirely in Rust, designed to operate without the need for API keys or external vector databases. Josh Clemm, a figure associated with the project, shared the news, stating, > "Open sourcing something fun from @Dropbox: Witchcraft. It's a local search engine built in Rust with no API keys or vector DB required." This development underscores Dropbox's continued investment in high-performance, privacy-focused local computing solutions.
Witchcraft is a reimplementation of Stanford's XTR-Warp semantic search engine, optimized for client-side deployment using a single-file SQLite database for storage. The engine leverages a ColBERT/late interaction style retrieval mechanism, enabling rapid search capabilities. Performance benchmarks indicate impressive speeds, with the system achieving a 21ms p.95 end-to-end search latency on the NFCorpus dataset, demonstrating efficiency that surpasses the original XTR-WARP on server-class hardware.
The project is particularly tailored for use with coding agents, offering a robust local solution for semantic search. An example application, "Pickbrain," is included, which indexes session transcripts from AI coding tools like Claude Code and OpenAI Codex. This functionality allows developers to quickly search through their past coding interactions, providing an efficient way to retrieve specific information or resume previous work.
Dropbox has increasingly adopted Rust for performance-critical components of its infrastructure, including its Magic Pocket storage system and the Nucleus sync engine. This strategic shift towards Rust is driven by the language's benefits in performance, type safety, and developer productivity, as noted in various company discussions. The open-sourcing of Witchcraft, under the Apache License, Version 2.0, aligns with this broader commitment to leveraging Rust for innovative solutions.
