Plotting Agent (Step 2)

Faithful implementation of the Plotting Agent from PaperOrchestra (Song et al., 2026, arXiv:2604.05018, §4 Step 2 and App. F.1 p.45).

Cost: ~20–30 LLM calls. The paper uses PaperBanana (Zhu et al., 2026) as the default backbone with a closed-loop VLM-critique refinement. This skill expresses that loop in host-agent terms: you (the host agent) generate matplotlib code with your own LLM, render via your Bash/Python tool, optionally critique the rendered PNG with your vision model, redraw, and finally caption.

Inputs

• workspace/outline.json

  — specifically the

  plotting_plan

  array

• workspace/inputs/idea.md

  and

  workspace/inputs/experimental_log.md

  — the source data

• workspace/inputs/figures/

  — optional pre-existing figures (

  PlotOn

  mode)

Outputs

• workspace/figures/<figure_id>.png

  — one PNG per

  plotting_plan

  entry (300 DPI, sized to the requested aspect ratio)

• workspace/figures/captions.json

  —

  {figure_id: caption_text}

  map

Workflow

Per figure (executed independently per

figure_id

)

1. Read the figure spec from

   outline.json

   :

   {
     "figure_id": "fig_main_results",
     "title": "Main Results on Dataset X",
     "plot_type": "plot",
     "data_source": "experimental_log.md",
     "objective": "Visual summary (Grouped Bar Chart) demonstrating ...",
     "aspect_ratio": "5:4"
   }
   

2. Few-shot retrieval (visual planning): pick the matching pattern from

   references/chart-patterns.md

   (for

   plot_type=="plot"

   ) or

   references/diagram-patterns.md

   (for

   plot_type=="diagram"

   ).

3. Extract data: parse

   idea.md

   and/or

   experimental_log.md

   (

   data_source

   field tells you which) to obtain the numeric values or conceptual entities the figure needs. For

   experimental_log.md

   , the

   ## 2. Raw Numeric Data

   section contains markdown tables.

4. Render:

   If

   PAPERBANANA_PATH

   is set — use the PaperBanana backbone (Zhu et al., 2026). It runs a Retriever → Planner → Stylist → Visualizer → Critic loop and is especially good for

   plot_type == "diagram"

   . See

   references/paperbanana-cookbook.md

   for setup (needs a Gemini API key).

   python skills/plotting-agent/scripts/paperbanana_render.py \
       --figure-id <figure_id> \
       --caption   "<objective from figure spec>" \
       --content-file workspace/inputs/idea.md \
       --task      <diagram|plot> \
       --aspect-ratio <aspect_ratio> \
       --out       workspace/figures/<figure_id>.png
   

   Otherwise — write a matplotlib script and run it via your Bash tool, or use the bundled helper:

   python skills/plotting-agent/scripts/render_matplotlib.py \
       --spec spec.json \
       --out workspace/figures/<figure_id>.png
   

   The script must apply the academic style from

   chart-patterns.md

   , use the correct pixel size from

   aspect-ratios.md

   , save at 300 DPI, and call

   plt.close()

   after

   savefig

   .

5. VLM critique loop (optional, only if your host has vision):

   • Reload the rendered PNG as a multimodal input to your LLM.
   • Critique it against the figure's

     objective

     from the outline. Look for: visual artifacts, mislabeled axes, illegible text, color clashes, misleading scaling, missing legend, overlapping labels.
   • If problems are found, regenerate the matplotlib script with corrections and re-render. Cap at 3 critique iterations per figure.
   • This is the closed-loop refinement step the paper inherits from PaperBanana. See

     references/plotting-pipeline.md

     for the full loop description.
   • If your host has no vision input, skip this step entirely. The figure will still render correctly, just without iterative refinement.

6. Generate the caption using the verbatim Caption Generation prompt at

   references/caption-prompt.md

   . Inputs to the caption prompt:

   • task_name

     — the section the figure belongs to (e.g., "Methodology", "Experiments")

   • raw_content

     — the surrounding section text (or content_bullets from the section_plan if the section isn't drafted yet)

   • description

     — the

     objective

     field from the figure spec

   • figure_desc

     — a 1-sentence description of what the rendered figure actually shows (from your VLM critique pass, or from the script's plan if no vision)

   Write the caption to

   workspace/figures/captions.json

   keyed by

   figure_id

   . Captions must NOT contain

   Figure N:

   or

   Caption N:

   prefixes — the LaTeX template handles numbering. Plain text only, no markdown.

Conceptual diagrams

For

plot_type == "diagram"

, prefer PaperBanana when available — its Retriever grounds the Planner in real published paper diagrams. If

PAPERBANANA_PATH

is unset, follow

references/diagram-patterns.md

. Patterns include block diagrams, system overviews, flowcharts, and algorithm-as-graph. The bundled helper:

python skills/plotting-agent/scripts/render_diagram.py \
    --spec diagram_spec.json \
    --out workspace/figures/<figure_id>.png

handles the simple cases (boxes-and-arrows). For complex Fig-1-style overview diagrams, write matplotlib patches code yourself.

Hard rules

• 300 DPI for every figure. Lower DPI gets rejected at the LaTeX compile step on conference templates.
• Aspect ratio is exact. The figure spec's

  aspect_ratio

  is one of 12 enumerated strings. Use the pixel targets in

  references/aspect-ratios.md

  .
• Hide top and right spines for plots. (Diagrams: no spines at all.)
• Muted academic colors only. The palette is in

  chart-patterns.md

  . Never use matplotlib defaults (too saturated for print).
• No 3D, no pie charts, no decorative visuals. The paper's evaluators penalize these.
• Every figure MUST have a caption in

  captions.json

  . The Section Writing Agent will fail-stop if a caption is missing for any figure referenced from the outline.
• No

  Figure N:

  prefix in captions — LaTeX adds it.
• Never describe data you didn't plot. The Plotting Agent must not hallucinate axes, baselines, or trends. Source-of-truth is

  experimental_log.md

  or

  idea.md

  .

Pre-existing figures (PlotOn mode)

If

workspace/inputs/figures/

is non-empty, check whether any pre-existing file matches a

figure_id

in the outline (by filename prefix). If so, copy it into

workspace/figures/

as-is and still generate a caption using the caption prompt. Only generate from scratch the figure_ids that have no pre-existing counterpart.

Resources

• references/caption-prompt.md

  — verbatim Caption Generation prompt from App. F.1

• references/plotting-pipeline.md

  — the full few-shot → render → critique → caption loop

• references/chart-patterns.md

  — matplotlib style + chart type recipes

• references/diagram-patterns.md

  — conceptual diagram recipes

• references/aspect-ratios.md

  — pixel targets for each of the 12 allowed ratios at 300 DPI

• references/paperbanana-cookbook.md

  — NEW PaperBanana setup, usage, cost notes, attribution

• scripts/render_matplotlib.py

  — render a JSON plot spec → PNG (matplotlib fallback)

• scripts/render_diagram.py

  — render a JSON diagram spec → PNG (matplotlib fallback)

• scripts/paperbanana_render.py

  — NEW PaperBanana backbone wrapper (reads

  PAPERBANANA_PATH

  from env)