# Contributing to Vega Datasets

Thank you for your interest in contributing to Vega Datasets! This repository serves as a centralized hub for datasets used across the Vega ecosystem (Vega, Vega-Lite, Altair).

We welcome contributions that enhance metadata, improve documentation, fix dataset issues, enhance infrastructure, or add new datasets that demonstrate visualization techniques. Each dataset should serve a clear purpose in showcasing visualization capabilities. 

All contributions are accepted under the [Project's license](https://github.com/vega/vega-datasets/blob/main/README.md#license). The Project abides by the Vega Organization's [code of conduct](https://github.com/vega/.github/blob/main/CODE_OF_CONDUCT.md) and [governance](https://github.com/vega/.github/blob/main/project-docs/GOVERNANCE.md).

## Dataset Contribution Guidelines

### General Principles

1. **Prioritize backward compatibility**: Generally avoid modifying existing datasets as other projects rely on them for tests and examples. Please review our [versioning guidelines](README.md#versioning) before making changes to ensure your contribution follows our versioning policy.

2. **Use open datasets**: All contributions must either use datasets available under open licenses (e.g., [CC0](https://creativecommons.org/public-domain/cc0/), [CC-BY](https://creativecommons.org/licenses/by/4.0/deed.en), [ODbL](https://opendatacommons.org/licenses/odbl/), Public Domain) or datasets with research-friendly licenses, government data, or other terms that permit redistribution with attribution. All submissions must include complete documentation of licensing terms, clear attribution information, and transparency about any usage limitations.

3. **Document thoroughly**: Provide detailed descriptions, sources, license information, and field definitions to ensure your data can be properly understood and utilized by users who may be unfamiliar with your dataset's domain. Good documentation preserves institutional knowledge about data provenance and processing methods. Each dataset in this repository is treated as a [data resource](https://datapackage.org/standard/data-resource/) within the [Data Package Standard v2](https://datapackage.org/standard/data-package/), which guides our documentation approach and ensures consistency.

**Important**: The BSD-3-Clause license of this repository applies to the package code and infrastructure, NOT to the datasets themselves. Each dataset maintains its original license.

### Proposing New Datasets

New datasets typically emerge organically from the needs of the Vega ecosystem. The path to contribution often follows one of these patterns:

* **Addressing visualization needs**: A limitation might be discovered while working with Altair, Vega, or Vega-Lite where an existing example or visualization technique could benefit from a specific dataset structure. These discussions may begin in any of the Vega ecosystem repositories.

* **Demonstrating new capabilities**: When new visualization features are developed, specialized datasets that effectively showcase these capabilities may be needed.

* **Improving examples**: Community members may identify gaps in documentation or tutorials that could be addressed with more representative datasets.

* **Exploring new domains**: Datasets that bring visualization capabilities to domains not yet represented in the Vega ecosystem may add value. When introducing data from a new domain:
   * Include domain-specific context in the documentation
   * Explain how the dataset enables interesting or novel visualizations
   * Highlight which aspects of the domain make it suitable for demonstration purposes

Before contributing a new dataset:

* **Connect with the community**: Either locate an existing issue discussing the need for such a dataset, or create one in the most relevant repository (Vega, Vega-Lite, Altair, or vega-datasets). This helps establish context for why the dataset is valuable.

* **Reference related discussions**: In your PR, reference any related discussions across repositories that demonstrate the need for this dataset.

The goal is to ensure that each dataset serves a clear visualization purpose. However, the process is often iterative and may cross multiple repositories.

### Data Generation Scripts

For datasets requiring processing:

- Place scripts in the `scripts/` directory and configuration files in `_data/`
- Include detailed documentation explaining parameters and processing steps
- Ensure reproducibility with deterministic outputs and fixed random seeds when applicable
- See `scripts/flights.py` as an example

## Metadata and Documentation

We follow the [Data Package Standard 2.0](https://datapackage.org/) with:

- Auto-generated `datapackage.json` and `datapackage.md` files
- Datasets in the `data/` directory
- Source metadata in `_data/datapackage_additions.toml`

### Metadata Requirements

The metadata system combines automatic inference with manual specification:

1. The `build_datapackage.py` script automatically infers basic metadata for each dataset (file format, size, data types)
2. You provide additional context and documentation in `_data/datapackage_additions.toml`
3. The build process merges these together, with manual entries taking precedence over inferred ones

For each dataset, add an entry to `_data/datapackage_additions.toml`:

```toml
[[resources]] # Path: example.json
path = "example.json"
description = """Detailed description of the dataset"""

# Schema section for documenting data fields/columns
# The system will infer basic column types automatically, but you should add descriptions
[resources.schema]
[[resources.schema.fields]]
name = "field1"  # Must match the actual column name in the dataset
description = "Description of what this column/field represents"
# Optional: Override the inferred type (see "Field Types" section below)
# type = "date"

[[resources.schema.fields]]
name = "field2"  # Must match the actual column name in the dataset
description = "Description of what this column/field represents"
# Optional: For categorical fields, list possible values
categories = ["category1", "category2", "category3"]

# For fields with numeric values, you can include statistics or other context if highly relevant
# [[resources.schema.fields]]
# name = "numericField"
# description = "mean: 0.5, range: [0, 1], etc."

# Data sources documentation
[[resources.sources]]
title = "Original Data Source Name"
path = "https://example.com/source-url"
# Optional: version information if available
# version = "1.0"

# License documentation
[[resources.licenses]]
name = "license-id"  # e.g., "CC-BY-4.0"
path = "https://example.com/license-url"
title = "Human-readable license name"
```

#### How the Metadata System Works

- **Automatic Inference**: For tabular files (CSV, TSV, JSON, Parquet, Arrow), the system automatically detects column names and data types
- **Manual Additions**: Use the TOML file to add descriptions, categories, source information, and licenses
- **Field Matching**: When documenting columns in `resources.schema.fields`, ensure the `name` field exactly matches the column name in your dataset
- **Precedence**: Your manual definitions in the TOML file will override any automatically inferred values
- **Complete Examples**: See existing entries in `_data/datapackage_additions.toml` for reference on documenting various types of datasets

#### Field Types

The Frictionless framework automatically infers field types by analyzing data values during the build process. You can explicitly override the inferred types by specifying a `type` field. Valid types according to the [Data Package Table Schema standard](https://datapackage.org/standard/table-schema/#field-types) are:

- **Text**: `string`
- **Numeric**: `number`, `integer`
- **Boolean**: `boolean`
- **Date/Time**: `date`, `time`, `datetime`, `year`, `yearmonth`, `duration`
- **Structured**: `object`, `array`, `list`
- **Spatial**: `geopoint`, `geojson`
- **Any**: `any` (accepts any type - used when a field contains mixed types or when type inference is not possible)

Example of overriding a field type:

```toml
[[resources.schema.fields]]
name = "date"
type = "datetime"  # Override inferred type
description = "Date and time of the observation"
```

Generate metadata files by running:

```bash
npm run build
```

## Development Setup

### Prerequisites

- Node.js (for npm)
- Python and [uv](https://github.com/astral-sh/uv) (for data processing scripts)

### Getting Started

1. Fork and clone the repository:
   ```
   git clone https://github.com/YOUR-USERNAME/vega-datasets.git
   cd vega-datasets
   npm install
   ```

2. (Optional) Set up Python environment:
   ```
   uv venv
   source .venv/bin/activate  # On Windows: .venv\Scripts\activate
   ```

> **Note**: `npm install` installs a pre-commit hook (via husky) that runs `uv run ruff` on staged `.py` files. Make sure `uv` is installed (see Prerequisites) or the hook will fail.

### Code Quality Checks

Run these checks before submitting (using `uv run` ensures correct tool versions from `uv.lock` are used):

```bash
# TOML formatting
uv run taplo fmt --check --diff

# Python linting and formatting
uv run ruff check
uv run ruff format --check
```

To automatically fix issues:
```bash
uv run taplo fmt
uv run ruff format
```

### Validating `datapackage.json` (optional, local)

After editing data files or the descriptor, you can validate the data
package end-to-end with pytest. Two tiers:

```bash
# Fast tier — file existence, declared bytes, git-blob SHA-1.
# Stdlib only, sub-second across all resources.
uv run --group dev pytest tests/

# Slow tier — frictionless schema and row validation per resource.
# Default is full read; flights-3m.parquet (~3M rows) takes minutes.
uv run --group dev pytest tests/ --runslow

# Slow tier with a row cap — matches what CI runs; lower for tighter iteration.
uv run --group dev pytest tests/ --runslow --limit-rows 250000
```

CI runs the slow tier with `--limit-rows 250000`: `flights_3m`'s ~3M
rows are sampled, every other resource is below the cap and validates
in full. The fast tier is implicitly covered too — `npm run build`
regenerates `datapackage.json` from on-disk data before the slow tier
runs, so any byte/hash drift would surface either there or in the slow
tier's schema validation.

Resources whose schema/row failures are known and non-actionable (for
example, `movies` whose schema is intentionally aspirational, or
`flights_200k_arrow` which frictionless can't parse) are listed in
[`_data/validate_datapackage.toml`](_data/validate_datapackage.toml).
The slow-tier test for each is marked `xfail(strict=True)`, so it does
not fail the run today — but if the upstream issue ever resolves, the
test flips XFAIL → XPASS and the run fails, prompting allowlist removal.

## Contributing Process

1. Create a branch:
   ```
   git checkout -b your-feature-branch
   ```

2. Make your changes following the guidelines above.

3. Run checks and build:
   ```
   uv run taplo fmt --check --diff && uv run ruff check && uv run ruff format --check
   npm run build
   ```

4. Commit and push:
   ```
   git commit -am "Add new dataset: description of dataset and changes"
   git push origin your-feature-branch
   ```

5. Open a pull request against the `main` branch.

6. Wait for CI checks to pass and respond to reviewer feedback.