Update Check — ONCE PER SESSION (mandatory) The first time this skill is used in a session, run the check-updates skill before proceeding.

• GitHub Copilot CLI / VS Code: invoke the

  check-updates

  skill.
• Claude Code / Cowork / Cursor / Windsurf / Codex: compare local vs remote package.json version.
• Skip if the check was already performed earlier in this session.

CRITICAL NOTES

1. To find the workspace details (including its ID) from workspace name: list all workspaces and, then, use JMESPath filtering
2. To find the item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace and, then, use JMESPath filtering

Spark Authoring — CLI Skill

Table of Contents

storage.azure.com

az login

az rest

--resource https://api.fabric.microsoft.com

az rest

curl

curl

az rest

/jobs/{jobType}/schedules

endDateTime

az rest

az rest

--resource

getLibrary()

libraryVariables

@pipeline().libraryVariables

/result

\n

format=ipynb

.ipynb

.py

# METADATA

Files/

?format=ipynb

--body '{}'

\n

Succeeded

Must/Prefer/Avoid

MUST DO

• Check for recent jobs BEFORE creating new notebook runs — Query job instances from last 5 minutes; if recent job exists, monitor it instead of creating duplicate
• Capture job instance ID immediately after POST — Store job ID before any other operations to enable proper monitoring
• Verify workspace capacity assignment before operations — Workspace must have capacity assigned and active
• When user provides a public data URL, follow the Public URL Data Ingestion policy — keep detailed behavior in the linked resource section to avoid drift/duplication
• Format notebook cells correctly — Each line in cell source array MUST end with

  \n

  to prevent code merging
• Use correct Livy session body format — Send a FLAT JSON with

  name

  ,

  driverMemory

  ,

  driverCores

  ,

  executorMemory

  ,

  executorCores

  . Do NOT wrap in

  {"payload": ...}

  or send only

  {"kind": "pyspark"}

  — that causes HTTP 500. Use valid memory values (28g, 56g, 112g, 224g). See Create Livy Session example below and SPARK-CONSUMPTION-CORE.md.

PREFER

• Poll job status with proper intervals — 10-30 seconds between polls; timeout after reasonable duration (e.g., 30 minutes)
• Check job history when POST response is unreadable — If POST returns "No Content" or unreadable response, query recent jobs (last 1 minute) before retrying
• Use Starter Pool for development — Development/testing workloads should use

  useStarterPool: true

• Use Workspace Pool for production — Production workloads need consistent performance with

  useWorkspacePool: true

• Enable lakehouse schemas during creation — Set

  creationPayload.enableSchemas: true

  for better table organization
• Implement idempotency checks — Prevent duplicate operations by checking existing state first

AVOID

• Never retry POST with same parameters — If you have a job ID, only use GET to check status; don't create duplicate job instances
• Don't skip capacity verification — Operations will fail if workspace capacity is paused or unassigned
• Avoid immediate POST retries on failures — Check for existing/active jobs first to prevent duplicates
• Don't create new runs if monitoring existing job — One job at a time; wait for completion before submitting new runs
• Don't hardcode workspace/lakehouse IDs — Discover dynamically via item listing or catalog search APIs

RULES — Read these first, follow them always

Rule 1 — Validate prerequisites before operations. Verify workspace has capacity assigned (see COMMON-CORE.md Create Workspace and Capacity Management) and resource IDs exist before attempting operations.

Rule 2 — Trust updateDefinition success. A

Succeeded

poll result from

updateDefinition

is sufficient confirmation that content and lakehouse bindings persisted. Do NOT call

getDefinition

after every upload — it is an async LRO that adds significant latency. Only use

getDefinition

for its intended purpose: reading current notebook content before making modifications.

Rule 3 — Prevent duplicate jobs and monitor execution properly. Before submitting new notebook run, ALWAYS check for recent job instances first (last 5 minutes). If recent job exists, monitor it instead of creating duplicate. After submission, capture job instance ID immediately and poll status - never retry POST. See SPARK-AUTHORING-CORE.md Job Monitoring for patterns.

Quick Start Examples

For detailed patterns, authentication, and comprehensive API usage, see:

• COMMON-CORE.md — Fabric REST API patterns, authentication, item discovery
• COMMON-CLI.md —

  az rest

  usage, environment detection, token acquisition
• SPARK-AUTHORING-CORE.md — Notebook deployment, lakehouse creation, job execution

Below are minimal quick-start examples. Always reference the COMMON- files for production use.*

Create Workspace & Lakehouse

# See COMMON-CORE.md Environment URLs and SPARK-AUTHORING-CORE.md for full patterns
cat > /tmp/body.json << 'EOF'
{"displayName": "DataEng-Dev"}
EOF
workspace_id=$(az rest --method post --resource "https://api.fabric.microsoft.com" \
  --url "https://api.fabric.microsoft.com/v1/workspaces" \
  --body @/tmp/body.json --query "id" --output tsv)

cat > /tmp/body.json << 'EOF'
{"displayName": "DevLakehouse", "type": "Lakehouse", "creationPayload": {"enableSchemas": true}}
EOF
lakehouse_id=$(az rest --method post --resource "https://api.fabric.microsoft.com" \
  --url "https://api.fabric.microsoft.com/v1/workspaces/$workspace_id/items" \
  --body @/tmp/body.json --query "id" --output tsv)

Organize Lakehouse Tables with Schemas

# See SPARK-AUTHORING-CORE.md Lakehouse Schema Organization for table organization patterns
# Create schemas for medallion architecture
spark.sql("CREATE SCHEMA IF NOT EXISTS bronze")
spark.sql("CREATE SCHEMA IF NOT EXISTS silver")
spark.sql("CREATE SCHEMA IF NOT EXISTS gold")

Create Livy Session

# See SPARK-CONSUMPTION-CORE.md for Livy session configuration and management
# IMPORTANT: Body MUST be flat JSON with memory/cores — do NOT wrap in {"payload": ...}
cat > /tmp/body.json << 'EOF'
{"name": "dev-session", "driverMemory": "56g", "driverCores": 8, "executorMemory": "56g", "executorCores": 8, "conf": {"spark.dynamicAllocation.enabled": "true", "spark.fabric.pool.name": "Starter Pool"}}
EOF
az rest --method post --resource "https://api.fabric.microsoft.com" \
  --url "https://api.fabric.microsoft.com/v1/workspaces/$workspace_id/lakehouses/$lakehouse_id/livyapi/versions/2023-12-01/sessions" \
  --body @/tmp/body.json

Livy Session Body — Common Mistakes

• ❌

  {"payload": {"kind": "pyspark"}}

  → HTTP 500 (wrong wrapper, missing required fields)
• ❌

  {"kind": "pyspark"}

  → HTTP 500 (missing

  driverMemory

  ,

  executorMemory

  , etc.)
• ✅ Flat JSON with

  name

  ,

  driverMemory

  ,

  driverCores

  ,

  executorMemory

  ,

  executorCores

  (and optionally

  conf

  with Starter Pool)

Spark Performance Configs

For detailed workload-specific configurations, see data-engineering-patterns.md Delta Lake Best Practices.

Quick reference:

# Write-heavy (Bronze): Disable V-Order, enable autoCompact
# Balanced (Silver): Enable V-Order, adaptive execution  
# Read-heavy (Gold): Vectorized reads, optimal parallelism
# See data-engineering-patterns.md for complete config tables

Variable Library in Notebooks

Use a Variable Library to centralize lakehouse names, workspace IDs, and feature flags.

# ✅ CORRECT — getLibrary() + dot notation
lib = notebookutils.variableLibrary.getLibrary("MyConfig")
lakehouse_name = lib.lakehouse_name
enable_logging = lib.enable_logging  # returns string "true"/"false"

# Boolean: compare as string (bool("false") is True in Python!)
if enable_logging.lower() == "true":
    print("Logging enabled")

# ❌ WRONG — .get() does not exist, causes runtime failure
# notebookutils.variableLibrary.get("MyConfig", "lakehouse_name")

Focus: Essential CLI patterns for Spark/data engineering development with intelligent routing to specialized resources. For comprehensive patterns, always reference COMMON-* files and resource documents.