Manaaki Whenua - Landcare Research

At Manaaki Whenua - Landcare Research, New Zealand’s Crown Research Institute for terrestrial environment and biodiversity, I worked on scaling internal tools that manage biological species research data.

Scaling BioTaNZ Data Exports

The BioTaNZ platform is a repository for biological specimen data collected in laboratory research. The critical bottleneck was in the data download functionality—the website was dynamically building CSV files from XML documents nested 4-5 levels deep, which was resource-intensive at scale.

The Solution:

1. Decoupled Backend Processing: Introduced automated preprocessing of incoming documents so data was converted to CSV format in the background rather than on client request.
2. SOLR Document Storage: Implemented SOLR for document indexing, providing automated updates when new data arrived.
3. Preprocessed Data Delivery: Changed client-side code to work with preprocessed CSV-formatted data instead of formatting on-the-fly.

Result: Increased CSV download throughput from 300 units to 30,000 units (rows in exported files)—a 100x improvement in capacity.

The graph above illustrates the dramatic improvement in throughput after implementing the decoupled processing architecture.