E2E Test Runner

Execute end-to-end test scenarios against running microservices with automatic debugging and reporting.

Usage

/e2e-test                    # Interactive - asks what to test
/e2e-test happy              # Happy path: create order flow
/e2e-test unhappy            # Unhappy path: out of stock, invalid data
/e2e-test cancel             # Cancel order flow
/e2e-test debug              # Just show service status and debug info
/e2e-test trace <corr-id>    # Trace request across services by CorrelationId
/e2e-test <custom scenario>  # Describe what you want to test

Input

• $ARGUMENTS

  - Test scenario to run or empty for interactive mode

Architecture Reference

Services

/api/*

/api/products

ProductService

/api/orders

Databases (PostgreSQL)

productdb

Product

Stock

StockReservation

OutboxMessage

InboxState

orderdb

Order

OrderItem

OutboxMessage

OutboxState

InboxState

notificationdb

ProcessedMessages

Stock Behavior (IMPORTANT)

Stock quantity is NOT decreased when orders are created. Instead:

• A

  StockReservation

  record is created with

  Status=0

  (Active)
• When order is cancelled, reservation changes to

  Status=1

  (Released)
• The

  Stock.Quantity

  field represents total inventory, not available inventory

Message Flow

Order API → [gRPC] → Product API (ReserveStock)
    ↓
OrderConfirmedEvent → [RabbitMQ] → Notification + Analytics

CRITICAL: Gateway-First Rule

ALL API calls in E2E tests MUST go through the Gateway. This is non-negotiable.

✓ CORRECT:  curl http://localhost:$GATEWAY_PORT/api/products
✗ WRONG:    curl http://localhost:$PRODUCT_PORT/api/products

Why:

• Gateway is the only entry point in production
• E2E tests must simulate real-world usage
• Tests the full stack: YARP proxy, correlation ID propagation, routing
• Catches Gateway-specific issues (routing, headers, timeouts)

When direct service access is allowed:

• debug

  scenario only - to compare Gateway vs direct response
• Troubleshooting when Gateway returns error but you need to verify backend works
• Never in

  happy

  ,

  unhappy

  ,

  cancel

  scenarios

Process

Phase 0: Service Mode Selection

ALWAYS start here. Use

AskUserQuestion

Question: "How should I manage services for this E2E test?"

Options:

1. Manual (Recommended) - "I'll run the services myself or they're already running"

   • User controls AppHost in their own terminal
   • Skill only discovers existing services
   • User responsible for starting/stopping

2. Automatic - "Start AppHost as a background process"

   • Skill starts

     dotnet run --project src/AppHost

     in background using

     run_in_background: true

   • Skill waits for services to be healthy
   • Skill offers to stop AppHost at the end using

     TaskStop

Based on selection:

• Manual mode: Proceed to Phase 1 (Environment Discovery)
• Automatic mode:
  1. Start AppHost in background:

     dotnet run --project src/AppHost
     

     Use

     run_in_background: true

     parameter
  2. Wait 15-30 seconds for services to start
  3. Run discovery to verify services are healthy
  4. If services not healthy after 60s, show logs and ask user what to do
  5. Remember the task_id - you'll need it for cleanup at the end

Phase 1: Environment Discovery

Run

./tools/e2e-test/discover.sh

1. Running services - ports dynamically assigned by Aspire
2. Database connection - PostgreSQL container and credentials
3. Message broker - RabbitMQ management URL
4. Health status - all service health endpoints

If services are not running:

• Manual mode: Inform user:

  Services not detected. Start with: dotnet run --project src/AppHost

• Automatic mode: Check AppHost logs for errors, report to user

Phase 2: Scenario Planning

Based on

$ARGUMENTS

happy

- Order Happy Path

All API calls via Gateway (

$GATEWAY_PORT

1. GET

   /api/products

   via Gateway → pick one with stock > 0
2. POST

   /api/orders

   via Gateway → create order (see API Contract below)
3. Verify: Order status = Confirmed (via Gateway GET

   /api/orders/{id}

   )
4. Verify: StockReservation created (Status=0) - DB query
5. Verify: Notification inbox has record (

   ProcessedMessages

   table) - DB query
6. Check logs for complete flow

API Contract - Create Order:

{
  "customerId": "guid",
  "customerEmail": "email@example.com",
  "items": [{
    "productId": "guid",
    "quantity": 2
  }]
}

unhappy

- Failure Scenarios

All API calls via Gateway (

$GATEWAY_PORT

1. Out of stock: POST

   /api/orders

   via Gateway with quantity > available
2. Invalid product: POST

   /api/orders

   via Gateway with non-existent productId
3. Invalid data: POST

   /api/orders

   via Gateway with missing required fields
4. Verify: Appropriate error responses from Gateway
5. Verify: No side effects (stock unchanged, no events) - DB queries

cancel

- Order Cancellation

All API calls via Gateway (

$GATEWAY_PORT

1. Create order first via Gateway (happy path steps 1-2)
2. POST

   /api/orders/{orderId}/cancel

   via Gateway (see API Contract below)
3. Verify: Order status = Cancelled (via Gateway GET

   /api/orders/{id}

   )
4. Verify: StockReservation status changed to 1 (Released) - DB query
5. Verify: Cancellation notification processed (

   OrderCancelledConsumer

   ) - DB query

API Contract - Cancel Order:

POST /api/orders/{orderId}/cancel
Content-Type: application/json

{"reason": "Customer requested cancellation"}  # Body is REQUIRED!

debug

- Service Debug Info

This is the only scenario where direct service access is allowed (for comparison/troubleshooting).

1. Show all service ports and URLs (Gateway + backend services)
2. Show database connection info
3. Show recent logs (last 20 lines per service)
4. Show message queue status (RabbitMQ)
5. Show gRPC connectivity status
6. Check service discovery configuration
7. (Optional) Compare Gateway response vs direct service response

trace <correlation-id>

- Distributed Request Tracing

1. Run

   ./tools/e2e-test/trace-correlation.sh <correlation-id>

2. Display chronologically sorted logs from all services
3. Highlight errors and warnings
4. Show service flow visualization

Options:

• --all-logs

  - Search all log files, not just latest

• --json

  - Output as JSON for further processing

Example:

/e2e-test trace 228617a4-175a-4384-a8e2-ade916a78c3f

Output shows:

• Service-colored log entries (gateway=cyan, order=green, product=yellow, etc.)
• Chronological order across all services
• Error highlighting in red, warnings in yellow

Phase 3: Execution

Execute scenario step by step. After each step:

1. Log the action taken
2. Verify expected outcome
3. If failure detected → STOP and consult user

REMEMBER: All API calls go through Gateway! (see Gateway-First Rule above)

Use these helpers:

# Service discovery - saves ports to .env file
./tools/e2e-test/discover.sh
source ./tools/e2e-test/.env

# API calls - ALWAYS use Gateway port!
curl -s "http://localhost:$GATEWAY_PORT/api/products" | jq '.'
curl -s "http://localhost:$GATEWAY_PORT/api/orders" | jq '.'
curl -s -X POST "http://localhost:$GATEWAY_PORT/api/orders" -H "Content-Type: application/json" -d '...'

# WRONG - never call services directly in E2E tests:
# curl -s "http://localhost:$PRODUCT_PORT/api/products"  # ✗ DON'T DO THIS
# curl -s "http://localhost:$ORDER_PORT/api/orders"      # ✗ DON'T DO THIS

# Database queries - get password first, then query
PG_PASS=$(docker exec <container> printenv POSTGRES_PASSWORD)
docker exec -e PGPASSWORD="$PG_PASS" <container> psql -U postgres -d <db> -c '<SQL>'

# Example:
PG_PASS=$(docker exec postgres-4cdf07e3 printenv POSTGRES_PASSWORD)
docker exec -e PGPASSWORD="$PG_PASS" postgres-4cdf07e3 psql -U postgres -d productdb -c 'SELECT * FROM "StockReservation";'

# Log inspection
./tools/e2e-test/logs.sh <service> [lines]

# Trace correlation ID
./tools/e2e-test/trace-correlation.sh <correlation-id>

Finding Service Ports Manually

If

discover.sh

# List all dotnet processes with ports
lsof -i -P -n | grep -E "EShop\.(Ord|Pro|Gat)" | grep LISTEN

# Typical output:
# EShop.Ord 45956  ... TCP 127.0.0.1:49814 (LISTEN)  <- Order API HTTP
# EShop.Pro 45955  ... TCP 127.0.0.1:49815 (LISTEN)  <- Product API HTTP
# EShop.Gat 45954  ... TCP 127.0.0.1:49818 (LISTEN)  <- Gateway HTTP

Phase 4: Reporting

Generate structured report:

═══════════════════════════════════════════════════════
  E2E TEST REPORT: <scenario name>
═══════════════════════════════════════════════════════

ENVIRONMENT
  Gateway:      http://localhost:XXXXX ✓  ← All API calls go here
  PostgreSQL:   localhost:XXXXX ✓
  RabbitMQ:     localhost:XXXXX ✓

  Backend services (for reference only):
    Order API:    http://localhost:XXXXX ✓
    Product API:  http://localhost:XXXXX ✓

SCENARIO: <description>

STEPS EXECUTED
  [✓] Step 1: Get products
      → Found 10 products, selected "Cable Management Kit" (stock: 100)

  [✓] Step 2: Create order
      → POST /api/orders → 201 Created
      → OrderId: abc-123-def

  [✓] Step 3: Verify order status
      → DB: Order.Status = 1 (Confirmed)

  [✗] Step 4: Verify stock decreased
      → Expected: 98, Actual: 100
      → FAILURE: Stock not reserved

LOGS (relevant entries)
  [Order.API 21:00:10] Creating order for customer X
  [Order.API 21:00:10] ERROR: gRPC call failed - No address resolver

DIAGNOSIS
  Problem: gRPC service discovery not configured
  Location: src/Common/EShop.ServiceClients/Extensions/ServiceCollectionExtensions.cs
  Fix: Add .AddServiceDiscovery() to gRPC client registration

RESULT: FAILED (Step 4)
═══════════════════════════════════════════════════════

Error Handling

When an error blocks the scenario:

1. STOP execution immediately
2. Gather diagnostic info:
   • Recent logs from affected service
   • Database state
   • Error response details
3. Present to user with options:

⚠️  Test blocked by error at Step X

Error: <description>
Service: <service name>
Log excerpt:
  <relevant log lines>

Options:
1. Attempt to fix the issue (I'll suggest a fix)
2. Skip this step and continue
3. Abort test and show partial report
4. Debug mode - show all diagnostic info

Wait for user decision before proceeding.

Key Validation Points

Note: All API calls in these validation points go through Gateway.

Order Creation (Happy Path)

POST $GATEWAY_PORT/api/orders

status: "Confirmed"

GET $GATEWAY_PORT/api/orders/{id}

status: "Confirmed"

SELECT * FROM "Order" WHERE "Id"='X'

Status = 1

SELECT * FROM "StockReservation" WHERE "OrderId"='X'

Status=0

SELECT "Quantity" FROM "Stock"

SELECT * FROM "OutboxMessage"

SELECT * FROM "ProcessedMessages"

OrderConfirmedConsumer

Order Cancellation

POST $GATEWAY_PORT/api/orders/{id}/cancel

status: "Cancelled"

GET $GATEWAY_PORT/api/orders/{id}

status: "Cancelled"

SELECT * FROM "Order" WHERE "Id"='X'

Status = 3

SELECT * FROM "StockReservation" WHERE "OrderId"='X'

Status=1

SELECT * FROM "ProcessedMessages"

OrderCancelledConsumer

Stock Low Alert

StockLowEvent

SELECT * FROM "ProcessedMessages"

StockLowConsumer

Log Patterns to Verify

Creating order for customer

Publishing OrderConfirmedEvent

ReserveStock request received

Stock reserved successfully

Consuming OrderConfirmedEvent

Sending email to

RabbitMQ Diagnostics

Use

./tools/e2e-test/rabbitmq.sh

./tools/e2e-test/rabbitmq.sh status      # Overview
./tools/e2e-test/rabbitmq.sh queues      # Queue status with message counts
./tools/e2e-test/rabbitmq.sh connections # Active service connections
./tools/e2e-test/rabbitmq.sh consumers   # Consumer registrations
./tools/e2e-test/rabbitmq.sh messages    # Pending message analysis

RabbitMQ Validation Points

Expected Queues (after first message)

order-confirmed     - OrderConfirmedEvent consumers
order-rejected      - OrderRejectedEvent consumers
order-cancelled     - OrderCancelledEvent consumers
stock-low           - StockLowEvent consumers

gRPC Diagnostics

Use

./tools/e2e-test/grpc.sh

./tools/e2e-test/grpc.sh status      # Port and connectivity check
./tools/e2e-test/grpc.sh list        # List services (requires grpcurl)
./tools/e2e-test/grpc.sh test        # Test gRPC calls
./tools/e2e-test/grpc.sh discovery   # Service discovery configuration

gRPC Services

ProductService

product.proto

GetProducts

ReserveStock

ReleaseStock

gRPC Validation Points

grpc.sh status

grpc.sh discovery

AddServiceDiscovery()

Cleanup After Testing

After completing tests, offer cleanup options based on the mode selected in Phase 0:

Ask User About Cleanup

Manual mode:

Test completed. Cleanup options:

1. Keep everything running (for further testing)
2. Reset test data (purge queues, clear orders)

Note: Stop services manually with Ctrl+C in your AppHost terminal.
What would you like to do?

Automatic mode (AppHost started by skill):

Test completed. Cleanup options:

1. Keep AppHost running (for further testing)
2. Stop AppHost (terminate background process)
3. Reset test data only (keep services running)
4. Full cleanup (stop AppHost + reset data)

What would you like to do?

Use

TaskStop

task_id

Cleanup Commands

Use

./tools/e2e-test/cleanup.sh

./tools/e2e-test/cleanup.sh status    # See what needs cleaning
./tools/e2e-test/cleanup.sh data      # Clear test orders, reservations
./tools/e2e-test/cleanup.sh queues    # Purge RabbitMQ queues
./tools/e2e-test/cleanup.sh logs      # Remove logs older than 7 days
./tools/e2e-test/cleanup.sh env       # Remove generated .env
./tools/e2e-test/cleanup.sh services  # Kill all running EShop services
./tools/e2e-test/cleanup.sh all       # Full cleanup (except services)

Kill Services

Use

./tools/e2e-test/kill-services.sh

./tools/e2e-test/kill-services.sh           # Show running services
./tools/e2e-test/kill-services.sh kill      # Kill with confirmation
./tools/e2e-test/kill-services.sh --force   # Kill without confirmation

This script finds and terminates all processes matching:

• EShop.*

  (AppHost, common libs)

• Order.API

  ,

  Products.API

  ,

  Gateway.API

• Notification.API

  ,

  Analytics.API

• DatabaseMigration

Or manually:

# Stop Aspire (if started by this session)
# Note: AppHost runs in foreground, Ctrl+C stops it

# Reset databases (clear test orders, keep products)
./tools/reset-db.sh

# Purge RabbitMQ queues (remove stuck messages)
./tools/e2e-test/rabbitmq.sh purge <queue-name>

Test Data Cleanup SQL

-- Clear orders (orderdb)
DELETE FROM "OrderItem";
DELETE FROM "Order";
DELETE FROM "OutboxMessage";
DELETE FROM "OutboxState";
DELETE FROM "InboxState";

-- Clear stock reservations (productdb)
DELETE FROM "StockReservation";

-- Note: Stock.Quantity doesn't need reset - it's not modified by orders

-- Clear processed messages (notificationdb) - note plural "Messages"
DELETE FROM "ProcessedMessages";

When to Clean Up

Important Notes

• Never auto-cleanup without asking user
• Preserve logs - useful for debugging issues found during tests
• Database reset uses

  ./tools/reset-db.sh

  which re-seeds products
• RabbitMQ queues auto-recreate when services restart
• Automatic mode cleanup - if you started AppHost in background, ALWAYS offer to stop it at the end using

  TaskStop

  with the saved

  task_id

• Manual mode - user manages services, don't attempt to stop them

Files Reference

./tools/e2e-test/discover.sh

./tools/e2e-test/db-query.sh

./tools/e2e-test/logs.sh

./tools/e2e-test/api.sh

./tools/e2e-test/rabbitmq.sh

./tools/e2e-test/grpc.sh

./tools/e2e-test/trace-correlation.sh

./tools/e2e-test/cleanup.sh

./tools/e2e-test/kill-services.sh

./tools/reset-db.sh

src/Services/*/logs/*.log

http://localhost:PORT/swagger

Technical Notes

StockReservation Table Schema

• ReservedAt

  - timestamp when reservation was made

• ExpiresAt

  - when reservation expires

• Status

  : 0 = Active, 1 = Released

Process Names in lsof

Aspire services use truncated names:

• EShop.Ord

  /

  Order.API

• EShop.Pro

  /

  Products.

• EShop.Gat

  /

  Gateway.A

The

discover.sh