Data Toolkit — November 13, 2026

Twenty-seven datasets, APIs and data tools this month, four datasets and four APIs to build on and nineteen libraries and engines to build with, a mix of the defaults and things that shipped in the last six months.

📚 Public Datasets

Open Food Facts

Barcode-level food, cosmetics and pet-food records, rebuilt nightly. The licence is ODbL for the database (attribution, and share-alike on adapted versions) with CC BY-SA on product images. Grab the gzipped CSV (about 0.9 GB compressed, 9 GB unpacked for food), the JSONL dump, a MongoDB dump, or the Parquet copy on Hugging Face. Good raw material for nutrition-score experiments and label parsers. Pricing: free (ODbL, images CC BY-SA).

NYC TLC Trip Record Data

Monthly Parquet files of yellow, green and for-hire-vehicle trips: pickup and drop-off zones and times, itemized fares, payment type, and since 2025 a cbd_congestion_fee column. January through September 2026 are posted, running about two months behind. The page states no licence, only accuracy disclaimers, so check terms before redistributing. It is the standard teaching dataset for anything that needs hundreds of millions of rows without scraping. Pricing: free, no registration.

Wikimedia Structured Contents (English and French Wikipedia)

Every article of both editions, already parsed into JSON-shaped Parquet with infoboxes, sections, tables, parsed references and citation-credibility signals, 44 GiB in total. Refreshed on 2026-05-19 with the parsed tables and reference scoring. Load it with load_dataset("wikimedia/structured-wikipedia", "enwiki_namespace_0"). The licence is CC BY-SA 4.0, so derived datasets carry share-alike. Pricing: free (CC BY-SA 4.0).

OpenAlex snapshot

About 626 million scholarly records (works, authors, sources, institutions, topics) in the September 2026 release, as JSON Lines (about 745 GB compressed) or Parquet (about 770 GB). The bucket LICENSE.txt is CC0. Pull it anonymously with aws s3 sync s3://openalex/data/parquet . --no-sign-request; the public snapshot refreshes quarterly. Daily snapshots need a paid plan. Pricing: free (CC0), no AWS account needed.

🔌 Free & Open APIs

National Weather Service API

GET https://api.weather.gov/points/{lat},{lon} returns the grid, then /gridpoints/.../forecast and /alerts/active?area=KS give forecasts and live alerts. No key; a descriptive User-Agent is required, and the rate limit is unpublished ("reasonable", retry after about 5 seconds). US only. Open data, free for any purpose, with an OpenAPI spec at /openapi.json. Pricing: free, no key.

USGS Earthquake Catalog (FDSN event service)

/fdsnws/event/1/query?format=geojson&starttime=... searches the global catalog and returns QuakeML, GeoJSON, CSV or KML, with a hard cap of 20,000 events per query (more gets a 400). No key and no published rate limit. USGS asks apps that just display recent quakes to use its static real-time GeoJSON feeds instead. Pricing: free, no key.

Frankfurter

Reference exchange rates for 219 currencies and 39 other assets from 111 central banks and official sources, with daily history back to 1948. v2 is current (GET /v2/rates?base=EUR, plus CSV and NDJSON output); v1 is deprecated but stays up. No key, no daily quota, only abuse throttling, and it is open source so you can self-host for heavy use. These are reference rates, not tick data. Pricing: free, no key.

OpenAlex API

The same scholarly graph as the snapshot, queryable: GET https://api.openalex.org/works?search=.... Without a key you get 10 requests per second; a free key raises that to 30 per second, gives a larger daily budget of $1 per day, and is needed for cursor paging past page one. Semantic search is capped at 1 request per second. Reach for it when you need a few thousand records, not 626 million. Pricing: free tier, key optional, paid plans for higher limits.

⚙️ Data Engineering & Pipelines

Dagster

An orchestrator built around assets (the tables and files you want to exist) rather than tasks, with lineage and partitions in the UI. Version 1.13.30 shipped on 2026-11-12, so it moves fast. Start with uv add dagster dagster-webserver dagster-dg-cli. Pricing: free (Apache 2.0); hosted Dagster+ is paid.

DuckLake

A lakehouse table format that keeps Parquet files in object storage and puts all metadata in an ordinary SQL database (DuckDB, SQLite, Postgres or MySQL) instead of manifest files. Version 1.0 landed in April 2026 with a backward-compatibility promise. In DuckDB: INSTALL ducklake; ATTACH 'ducklake:metadata.ducklake' AS lake (DATA_PATH 'data/');. Pricing: free (MIT).

Debezium

Log-based change data capture: it reads the write-ahead log of Postgres, MySQL, MariaDB, SQL Server, Oracle or MongoDB and emits each row change as an event, usually through Kafka Connect or Debezium Server. 3.7.1.Final is the current stable line, and the 3.8 alphas are out. Established, and still the default when a nightly dump is too slow. Pricing: free (Apache 2.0).

SQLMesh

A SQL transformation framework that plans changes before applying them: it diffs your models, shows which downstream tables rebuild, and uses virtual environments so a dev branch does not copy data. Now a Linux Foundation project, release v0.236.4 on 2026-10-22. pip install sqlmesh. Pricing: free (Apache 2.0).

📊 Analytics & Notebooks

pandas

Still the DataFrame everyone starts with. pandas 3.0 came out 2026-01-21, 3.1.0 followed on 2026-10-22 and 3.1.1 on 2026-11-04. If your code has not been run on 3.x yet, do that before an upgrade surprises you. pip install pandas. Pricing: free (BSD-3).

Ibis

One DataFrame-style Python API that compiles to the SQL of your engine, whether DuckDB on a laptop or BigQuery, Snowflake or Postgres in production. You write the analysis once and swap the backend. Version 12.0.0 was released 2026-02-07. pip install 'ibis-framework[duckdb]'. Pricing: free (Apache 2.0).

Apache DataFusion

A Rust query engine on Arrow with SQL and DataFrame front ends, written to be embedded; many newer query systems are built on it. Python users get pip install datafusion and can query Parquet or CSV in place. Choose it when you want an engine inside your own tool rather than a database to run. Pricing: free (Apache 2.0).

🕸️ Scraping & Collection

Trafilatura

Pulls the main text, metadata and comments out of a web page and discards menus and boilerplate, with sitemap, feed and focused-crawl helpers. Its crawler follows the Robots Exclusion Standard where applicable and waits between requests to the same host. pip install trafilatura; v2.3.1 shipped 2026-10-06. Pricing: free (Apache 2.0).

Crawl4AI

An async crawler that renders pages in a browser and returns clean Markdown or structured JSON for RAG pipelines. Robots handling is opt-in: set check_robots_txt=True in the config (default is off), set a user_agent that identifies you, and use the delay settings on arun_many(). v0.9.5 shipped 2026-10-14. pip install crawl4ai. Pricing: free (Apache 2.0).

Docling

Converts PDF, DOCX, PPTX, XLSX, HTML, images and more into Markdown or JSON, keeping layout, reading order and table structure, with OCR for scans. It runs locally, so documents never leave the machine. pip install docling; v2.150.0 shipped 2026-11-10. It processes files you already have; it does not fetch pages. Pricing: free (MIT).

✅ Data Quality & Catalogs

Pandera

Declare a DataFrame's columns, dtypes and checks once, then validate in tests or at pipeline boundaries, across pandas, Polars and other backends. Version 0.36.0 came out 2026-11-10. pip install pandera. Lighter than a full expectations framework. Pricing: free (MIT).

OpenMetadata

A catalog, lineage graph and glossary in one self-hosted app, with connectors for warehouses, BI tools and orchestrators. 2.0.0 was released 2026-08-24 after the 1.12 line, and 2.0.7 followed on 2026-10-27. Run it with the Docker Compose quickstart in the docs. Pricing: free (Apache 2.0); managed hosting is paid.

OpenLineage

An open spec for emitting run, job and dataset events, plus integrations for Airflow, Spark, dbt and Flink, so a catalog can draw lineage without each tool inventing its own format. Release 1.55.0 on 2026-10-30. Pair it with a backend such as OpenMetadata or Marquez. Pricing: free (Apache 2.0).

🔍 Vector & Search

pgvector

A Postgres extension that adds a vector column type, with HNSW and IVFFlat indexes and L2, inner-product and cosine distance. If your rows already live in Postgres, CREATE EXTENSION vector; is often the whole vector-database decision. Latest tag v0.8.7. Pricing: free (PostgreSQL License).

FAISS

Meta's library for similarity search over dense vectors, from exact flat indexes to compressed and GPU variants. It is a library, not a server, so you own persistence and filtering. Version 1.16.0 shipped 2026-11-10. pip install faiss-cpu. Pricing: free (MIT).

Typesense

A typo-tolerant search engine with faceting, geosearch and vector or hybrid queries, run as a single binary or Docker container. v30.2 shipped 2026-04-19. The server is GPL-3.0, so read the licence before you fork and redistribute it. Pricing: free (GPL-3.0); Typesense Cloud is paid.

📈 Visualization & Dashboards

Metabase

A self-hostable BI tool where analysts ask questions through a point-and-click builder or SQL and publish dashboards. v0.65.4 shipped 2026-11-13. Run it with docker run -p 3000:3000 metabase/metabase. The open-source edition is AGPL; embedding and some admin features sit in commercial editions. Pricing: free (AGPL open-source edition); paid tiers from the vendor.

Observable Plot

A JavaScript library for exploratory charts built from marks and scales, working on plain arrays of objects. Version 0.6.19 shipped 2026-11-05. Install with npm install @observablehq/plot. Good for a quick look at a new dataset in the browser. Pricing: free (ISC).

Altair

Declarative statistical charts in Python: describe the encodings, and Vega-Lite does the drawing, with interactive selections in notebooks. 6.3.0 shipped 2026-09-15. pip install altair. Pricing: free (BSD-3).

---

27 entries this month. Every dataset and API lists its licence and limits — check them before you build. Found a dead link or have something to add? Reply to the email.