Polars Skill

Polars DataFrame library for high-performance data manipulation in Python. Covers lazy/eager execution, expressions, I/O (CSV, Parquet, JSON, database), aggregations, joins, string/datetime operations, pandas/NumPy interop, and performance optimization. Use when working with Polars DataFrames, migrating from pandas, reading Parquet files, or optimizing data pipeline performance.

Comprehensive skill for high-performance data manipulation with Polars. Use decision trees below to find the right guidance, then load detailed references.

What is Polars?

Polars is a fast DataFrame library for Python (and Rust):

• Fast: Written in Rust, optimized for modern CPUs with SIMD and parallelism
• Lazy Evaluation: Build query plans that get optimized before execution
• Expressive: Powerful expression API for complex transformations
• Memory Efficient: Columnar format, streaming for larger-than-memory data
• No Dependencies: Pure Rust core, no NumPy/Pandas required

Version Notes

This skill targets Polars 1.x (tested with 1.37.1). Key changes from 0.x:

• apply

  renamed to

  map_elements

  (0.19+)

• groupby

  renamed to

  group_by

  (0.19+)

• melt

  renamed to

  unpivot

  (1.0+)
• Streaming engine improvements in 1.x

• pl.Utf8

  is now

  pl.String

  (1.0+, Utf8 still works as alias)

How to Use This Skill

Reference File Structure

Each topic in

./references/

quickstart.md

dataframes-series.md

io-data.md

expressions.md

aggregations-grouping.md

joins-concat.md

strings-datetime-categorical.md

performance.md

interop.md

gotchas.md

Reading Order

1. New to Polars? Start with

   quickstart.md

   then

   expressions.md

2. Coming from Pandas? Read

   quickstart.md

   ,

   expressions.md

   , then

   interop.md

3. Performance issues? Check

   performance.md

   first

Quick Decision Trees

"I need to get started"

Getting started?
├─ Install Polars → ./references/quickstart.md
├─ Create first DataFrame → ./references/quickstart.md
├─ Understand lazy vs eager → ./references/quickstart.md
├─ Learn expression syntax → ./references/expressions.md
└─ Coming from Pandas → ./references/interop.md

"I need to load or save data"

Loading/saving data?
├─ Read CSV file → ./references/io-data.md
├─ Read Parquet (recommended) → ./references/io-data.md
├─ Read JSON/NDJSON → ./references/io-data.md
├─ Read from database → ./references/io-data.md
├─ Read multiple files (glob) → ./references/io-data.md
├─ Write to file → ./references/io-data.md
└─ Larger-than-memory data → ./references/performance.md

"I need to filter or select data"

Filtering/selecting?
├─ Select columns by name → ./references/dataframes-series.md
├─ Select by pattern/regex → ./references/dataframes-series.md
├─ Select by data type → ./references/dataframes-series.md
├─ Filter rows by condition → ./references/dataframes-series.md
├─ Filter with multiple conditions → ./references/dataframes-series.md
├─ Handle null values → ./references/dataframes-series.md
└─ Add/modify columns → ./references/dataframes-series.md

"I need to aggregate or group data"

Aggregating data?
├─ Basic statistics (sum, mean, etc.) → ./references/aggregations-grouping.md
├─ Group by columns → ./references/aggregations-grouping.md
├─ Multiple aggregations → ./references/aggregations-grouping.md
├─ Window functions (over) → ./references/aggregations-grouping.md
├─ Rolling/moving averages → ./references/aggregations-grouping.md
├─ Cumulative operations → ./references/aggregations-grouping.md
└─ Ranking within groups → ./references/aggregations-grouping.md

"I need to combine DataFrames"

Combining data?
├─ Join two DataFrames → ./references/joins-concat.md
├─ Left/right/outer join → ./references/joins-concat.md
├─ Anti-join (not in) → ./references/joins-concat.md
├─ Concatenate vertically → ./references/joins-concat.md
├─ Pivot (long to wide) → ./references/joins-concat.md
└─ Unpivot/melt (wide to long) → ./references/joins-concat.md

"I need better performance"

Performance issues?
├─ Use lazy evaluation → ./references/performance.md
├─ Avoid row iteration → ./references/performance.md
├─ Reduce memory usage → ./references/performance.md
├─ Process large files → ./references/performance.md
├─ Optimize query plan → ./references/performance.md
└─ Common anti-patterns → ./references/performance.md

"Something isn't working"

Having issues?
├─ Type errors → ./references/gotchas.md
├─ Null handling → ./references/gotchas.md
├─ Expression context errors → ./references/gotchas.md
├─ String operations → ./references/strings-datetime-categorical.md
├─ Date parsing issues → ./references/strings-datetime-categorical.md
├─ Performance problems → ./references/gotchas.md
├─ Pandas migration issues → ./references/gotchas.md
├─ Memory errors → ./references/gotchas.md
└─ General troubleshooting → ./references/gotchas.md

File-First Execution in Research Workflows

Important: In data research pipelines (see

CLAUDE.md

The pattern:

1. Write transformation code to

   scripts/stage{N}_{type}/{step}_{task-name}.py

2. Execute via Bash with automatic output capture wrapper script
3. Validation results get automatically embedded in scripts as comments
4. If failed, create versioned copy for fixes

Closely read

agent_reference/SCRIPT_EXECUTION_REFERENCE.md

See:

• agent_reference/SCRIPT_EXECUTION_REFERENCE.md

  — Script execution protocol and format with validation

The examples below show Polars syntax. In research workflows, wrap them in scripts following the file-first pattern.

Quick Reference

Essential Import

import polars as pl
import polars.selectors as cs  # For column selection by type

Lazy vs Eager (One-Liner)

# Eager: immediate execution
df = pl.read_csv("data.csv")

# Lazy: deferred, optimized execution (preferred for large data)
lf = pl.scan_csv("data.csv")
df = lf.collect()  # Execute when ready

Core Expression Patterns

# Select columns
df.select("a", "b")
df.select(pl.col("a"), pl.col("b"))
df.select(pl.all().exclude("id"))

# Filter rows
df.filter(pl.col("a") > 10)
df.filter((pl.col("a") > 10) & (pl.col("b") == "x"))

# Add/modify columns
df.with_columns(
    (pl.col("a") * 2).alias("a_doubled"),
    pl.col("b").str.to_uppercase().alias("b_upper")
)

# Conditional column
df.with_columns(
    pl.when(pl.col("a") > 10)
      .then(pl.lit("high"))
      .otherwise(pl.lit("low"))
      .alias("category")
)

# Group and aggregate
df.group_by("category").agg(
    pl.col("value").sum().alias("total"),
    pl.col("value").mean().alias("average"),
    pl.len().alias("count")
)

Essential Functions

pl.col("name")

pl.lit(value)

pl.all()

pl.exclude("col")

pl.len()

pl.when().then().otherwise()

.alias("name")

.cast(pl.Int64)

Common Data Types

pl.Int64

pl.Int32

pl.Float64

pl.Float32

pl.String

pl.Utf8

pl.Boolean

pl.Date

pl.Datetime

pl.Duration

pl.Categorical

pl.List

pl.Struct

Quick Cheatsheet

# I/O
df = pl.read_csv/parquet/json("file")
lf = pl.scan_csv/parquet/ndjson("file")  # Lazy
df.write_csv/parquet/json("file")

# Selection
df.select("a", "b")
df.select(cs.numeric())  # By type

# Filtering
df.filter(pl.col("a") > 1)

# Aggregation
df.group_by("key").agg(pl.col("val").sum())

# Joining
df1.join(df2, on="key", how="left")

# Sorting
df.sort("col", descending=True)

# Lazy execution
lf.collect()  # Run query
lf.explain()  # Show plan

Topic Index

./references/quickstart.md

./references/quickstart.md

./references/quickstart.md

./references/dataframes-series.md

./references/dataframes-series.md

./references/dataframes-series.md

./references/io-data.md

./references/io-data.md

./references/io-data.md

./references/expressions.md

./references/expressions.md

./references/expressions.md

./references/aggregations-grouping.md

./references/aggregations-grouping.md

./references/aggregations-grouping.md

./references/joins-concat.md

./references/joins-concat.md

./references/joins-concat.md

./references/strings-datetime-categorical.md

./references/strings-datetime-categorical.md

./references/strings-datetime-categorical.md

./references/performance.md

./references/performance.md

./references/performance.md

./references/interop.md

./references/interop.md

./references/interop.md

./references/gotchas.md

./references/gotchas.md

./references/gotchas.md

./references/gotchas.md

./references/gotchas.md

./references/gotchas.md

./references/gotchas.md

Citation

When this library is used as a primary analytical tool, include in the report's Software & Tools references:

Vink, R. et al. Polars: Blazingly fast DataFrames [Computer software]. https://pola.rs/

Cite when: Polars is the core data processing engine for the analysis (typically always true in DAAF pipelines). Do not cite when: Only used for trivial file I/O in a script primarily using another tool.