Bulk Download Insider Trading Data: Nightly Exports vs Paginated API Calls

Compare paginated API calls and nightly exports to efficiently download insider trading data, and learn how to build a simple data pipeline with NDJSON.

If you need every insider trade since 2020, every Congress trade or every 13F holding, calling a REST API page by page is the slow way. Most data teams end up with a nightly bulk load instead. This guide compares the options and shows a simple pipeline.

Key takeaways

• Paginated endpoints are for apps that show a few rows at a time.
• Bulk endpoints return large filtered slices (up to 100,000 rows per request on API Pro).
• Nightly exports give you whole datasets as compressed files, ideal for warehouses and backtests.
• NDJSON (one JSON object per line) streams well and loads into almost any database.

Three ways to get the data

1. Paginated REST. GET /v1/insiders?ticker=NVDA&limit=100, then follow the next link. Perfect for an app page, slow for millions of rows.

2. Bulk requests. GET /v1/bulk/insiders?format=ndjson&from=2025-01-01 returns up to 50,000 rows (Premium) or 100,000 rows (API Pro) in one response, with the same filters as the normal endpoint. Good for weekly refreshes of one dataset.

3. Nightly exports. Every night each dataset is written to a gzipped NDJSON file. GET /v1/exports returns signed download links valid for one hour. This is how you mirror the whole Bourse catalogue.

Why NDJSON

Each line is a complete JSON record, so you can stream a file without loading it into memory, split it, and load it in parallel. Field names match the API exactly, so code written against the REST endpoints works on the files.

A nightly pipeline in 20 lines

```python import gzip, json, os, requests

KEY = os.environ["BOURSE_KEY"] H = {"Authorization": f"Bearer {KEY}"} files = requests.get("https://api.bourse.pl/v1/exports", headers=H).json()["data"]

for f in files: if f["dataset"] not in ("insiders", "congress", "funds13f"): continue raw = requests.get(f["url"]).content # signed link, no key needed rows = [json.loads(line) for line in gzip.decompress(raw).splitlines()] print(f["dataset"], len(rows), "rows") # write rows to Postgres, BigQuery, Snowflake or Parquet here ```

Schedule it with cron, Airflow or a GitHub Action. Because files are rebuilt every night, you can truncate and reload, or upsert by id.

Incremental updates

Between nightly loads, two options keep you current:

• Webhooks. New filings are POSTed to your endpoint every 5 minutes, signed with HMAC-SHA256. Insert them as they arrive.
• Date filters. Call ?from= with yesterday's date on the regular or bulk endpoint.

Which plan includes what

Summary

Use pages for apps, bulk for targeted refreshes and nightly exports for research and warehouses. Nightly exports and the commercial license come with Bourse API Pro; the API documentation lists every dataset you can export.

Bourse provides financial data and editorial information, parts of which are generated by AI. This page is for general education only. It is not personalised investment advice or a recommendation to buy or sell any financial instrument. Make your own decisions.