bulk-merge

Overview Link to heading

bulk-merge is a Rust CLI that ingests large bibliographic metadata dumps into PostgreSQL as usable, queryable tables.

Ambition Link to heading

Build the foundation for a universal open-library index that can handle hundreds of millions of records without breaking a sweat.

What’s novel Link to heading

  • High-performance ingestion using PostgreSQL’s COPY command.
  • Resumable import system for multi-gigabyte bibliographic dumps.
  • Dedicated schema management for diverse metadata sources.

Highlights Link to heading

  • LibGen-only ingestion
  • Dedicated tables per dump kind (fiction vs compact)
  • Resumable imports and incremental updates tracked in bm_meta
  • Ingest speed first: bulk load via COPY, create indexes after load
  • 1-to-1 field mapping from the MySQL dump to PostgreSQL columns (no semantic normalization yet)

Stats Link to heading

  • Project page: /projects/bulk-merge/
  • Primary language: Rust
  • Commits: 56
  • Created: 2026-04-13T23:06:29Z
  • Last updated: 2026-04-22T23:53:12Z