/deploy-frappe - Frappe HRMS Deployment

What This Command Does

Deploys code changes from the

production

IMPORTANT: Frappe Docker uses

--preload

TEST LOCALLY FIRST: Always test your changes in the local development environment before deploying to production. Use

/local-frappe

Quick Reference

no_cache=true

bench migrate

⚠️ CRITICAL: Docker Build Caching Issue

Problem discovered 2026-01-29: When pushing Python/API code changes, the build may use cached Docker layers and NOT include your new code.

When to Use

no_cache=true

no_cache=true

.py

apps.json

How to Deploy with No Cache

Option 1: Manual Workflow Dispatch (Recommended)

gh workflow run build-and-deploy.yml --repo Bebang-Enterprise-Inc/hrms -f no_cache=true

Option 2: Via GitHub UI

1. Go to https://github.com/Bebang-Enterprise-Inc/hrms/actions
2. Select "Build and Deploy Frappe HRMS"
3. Click "Run workflow"
4. Check

   no_cache

   checkbox ← IMPORTANT
5. Click "Run workflow"

Why This Happens

The Docker build uses layer caching. The HRMS app is cloned during build via

apps.json

Build time comparison:

• With cache: ~30 seconds (may miss code changes)
• Without cache: ~5 minutes (guaranteed fresh code)

Infrastructure (UPDATED 2026-01-29 - DOCKER SWARM)

i-026b7477d27bd46d6

samkarazi/bebang-erpnext-hrms:v15

hq.bebang.ph

ap-southeast-1

/home/ubuntu/frappe_docker

frappe

stack.yml

Swarm Services (9 total)

frappe_backend

frappe_frontend

frappe_websocket

frappe_queue-short

frappe_queue-long

frappe_scheduler

frappe_db

frappe_redis-cache

frappe_redis-queue

Volume Configuration

The site data is stored in external volumes:

bebang-hrms_sites

bebang-hrms_db-data

bebang-hrms_logs

bebang-hrms_redis-queue-data

Backups:

/home/ubuntu/backups/swarm-migration-20260129/

Deployment Methods

Method 1: PR Workflow (Recommended) - INCLUDES AUTO-CLEANUP

The

production

Note: The GitHub Actions workflow automatically cleans up old images after each successful deployment, keeping the 4 most recent versions. No manual cleanup needed when using this method.

# 1. Create feature branch (if not already on one)
/feature-branch my-api-change

# 2. Commit your changes
git add .
git commit -m "feat: Add employee_clearance API"

# 3. Deploy via PR (auto-merges when CI passes)
/pr-deploy --auto-merge

This creates a PR, enables auto-merge, and GitHub Actions builds and deploys when the PR merges.

See

/workflow

Method 1b: Manual Workflow Dispatch

For triggering a rebuild without merging (e.g., to force

no_cache

1. Go to https://github.com/Bebang-Enterprise-Inc/hrms/actions
2. Select "Build and Deploy Frappe HRMS"
3. Click "Run workflow"
4. Options:

   • no_cache

     : Build without Docker cache (USE THIS FOR CODE CHANGES)

   • skip_build

     : Skip image rebuild (deploy existing)

   • run_migrate

     : Run

     bench migrate

     after deploy

Method 2: AWS SSM Direct (Quick Deploy via Swarm Rolling Update)

Use when image is already built and you need to update services with zero downtime:

# Get AWS credentials
export AWS_ACCESS_KEY_ID=$(doppler secrets get AWS_ACCESS_KEY_ID --project bei-erp --config dev --plain)
export AWS_SECRET_ACCESS_KEY=$(doppler secrets get AWS_SECRET_ACCESS_KEY --project bei-erp --config dev --plain)
export AWS_DEFAULT_REGION=ap-southeast-1

# Rolling update all services (ZERO DOWNTIME) + CLEANUP OLD IMAGES
aws ssm send-command \
  --instance-ids "i-026b7477d27bd46d6" \
  --document-name "AWS-RunShellScript" \
  --parameters 'commands=[
    "echo === Updating services ===",
    "docker service update --image samkarazi/bebang-erpnext-hrms:v15 frappe_backend",
    "docker service update --image samkarazi/bebang-erpnext-hrms:v15 frappe_frontend",
    "docker service update --image samkarazi/bebang-erpnext-hrms:v15 frappe_websocket",
    "docker service update --image samkarazi/bebang-erpnext-hrms:v15 frappe_queue-short",
    "docker service update --image samkarazi/bebang-erpnext-hrms:v15 frappe_queue-long",
    "docker service update --image samkarazi/bebang-erpnext-hrms:v15 frappe_scheduler",
    "docker service ls",
    "echo === Cleaning up old images (keeping 4 newest) ===",
    "docker container prune -f",
    "IMAGES=$(docker images samkarazi/bebang-erpnext-hrms --format \"{{.ID}}\" | tail -n +5)",
    "if [ -n \"$IMAGES\" ]; then docker rmi $IMAGES 2>/dev/null || true; fi",
    "docker image prune -f",
    "df -h / | tail -1"
  ]' \
  --output json

Benefits of Swarm (migrated 2026-01-29):

• Zero-downtime rolling updates
• Automatic rollback on failure (

  docker service rollback <service>

  )
• No CSS 404 issues (containers fully recreated)

Check service status:

aws ssm send-command \
  --instance-ids "i-026b7477d27bd46d6" \
  --document-name "AWS-RunShellScript" \
  --parameters 'commands=["docker service ls"]' \
  --output json

Method 3: Manual Image Build (Recovery/Debug)

Use when GitHub Actions fails or you need custom build:

# 1. Clone frappe_docker
git clone https://github.com/frappe/frappe_docker.git
cd frappe_docker

# 2. Copy apps.json from BEI-ERP repo
cat > apps.json << 'EOF'
[
    {"url": "https://github.com/frappe/erpnext", "branch": "version-15"},
    {"url": "https://github.com/frappe/payments", "branch": "version-15"},
    {"url": "https://github.com/Bebang-Enterprise-Inc/hrms", "branch": "production"}
]
EOF
# Note: Repo is public, so no authentication needed for cloning

# 3. Build image
export APPS_JSON_BASE64=$(base64 -w 0 apps.json)
docker build \
  --no-cache \
  --build-arg FRAPPE_BRANCH=version-15 \
  --build-arg PYTHON_VERSION=3.11.6 \
  --build-arg NODE_VERSION=20.19.2 \
  --build-arg APPS_JSON_BASE64=$APPS_JSON_BASE64 \
  --file images/custom/Containerfile \
  --tag bebang/erpnext-hrms:v15-$(date +%Y%m%d) \
  --tag bebang/erpnext-hrms:v15 \
  .

# 4. Push to Docker Hub
docker login
docker push bebang/erpnext-hrms:v15-$(date +%Y%m%d)
docker push bebang/erpnext-hrms:v15

# 5. Deploy to server (via SSM - see Method 2)

Post-Deployment Steps

Automatic Image Cleanup (ALWAYS RUN)

After every deployment, clean up old images to prevent disk space issues.

The server has limited storage (50 GB). Each deployment creates a new ~2.5 GB image. Without cleanup, the disk fills up within days.

Policy: Keep the 4 most recent images, remove all older ones.

# Run this after every successful deployment
aws ssm send-command \
  --instance-ids "i-026b7477d27bd46d6" \
  --document-name "AWS-RunShellScript" \
  --parameters 'commands=[
    "echo \"=== Cleaning up old Docker images (keeping 4 newest) ===\"",
    "docker container prune -f",
    "IMAGES=$(docker images samkarazi/bebang-erpnext-hrms --format \"{{.ID}}\" | tail -n +5)",
    "if [ -n \"$IMAGES\" ]; then docker rmi $IMAGES 2>/dev/null || true; fi",
    "docker image prune -f",
    "echo \"=== Disk usage after cleanup ===\"",
    "df -h / | tail -1"
  ]' \
  --output json

What this does:

1. Removes exited containers (from previous deployments)
2. Lists all bebang-erpnext-hrms images, skips the 4 newest, removes the rest
3. Removes any remaining dangling images
4. Reports disk usage

Expected savings: ~2.5 GB per old image removed.

Run Migrations (if schema changed)

# Get the backend container ID
aws ssm send-command \
  --instance-ids "i-026b7477d27bd46d6" \
  --document-name "AWS-RunShellScript" \
  --parameters 'commands=["docker exec $(docker ps -qf name=frappe_backend) bench --site hq.bebang.ph migrate"]' \
  --output json

Clear Cache

aws ssm send-command \
  --instance-ids "i-026b7477d27bd46d6" \
  --document-name "AWS-RunShellScript" \
  --parameters 'commands=["docker exec $(docker ps -qf name=frappe_backend) bench --site hq.bebang.ph clear-cache"]' \
  --output json

Verify API

# Use frappe.ping (doesn't require auth)
curl https://hq.bebang.ph/api/method/frappe.ping
# Expected: {"message":"pong"}

# Or check login page
curl -s -o /dev/null -w "%{http_code}" https://hq.bebang.ph/login
# Expected: 200

Emergency Rollback

Option A: Swarm Service Rollback (Fastest)

Docker Swarm keeps previous service state for instant rollback:

# Rollback individual services
aws ssm send-command \
  --instance-ids "i-026b7477d27bd46d6" \
  --document-name "AWS-RunShellScript" \
  --parameters 'commands=[
    "docker service rollback frappe_backend",
    "docker service rollback frappe_frontend",
    "docker service rollback frappe_websocket",
    "docker service rollback frappe_queue-short",
    "docker service rollback frappe_queue-long",
    "docker service rollback frappe_scheduler",
    "docker service ls"
  ]' \
  --output json

Option B: Deploy Previous Image Tag

# Find previous tag from Docker Hub or deployment logs
# Tags are formatted: v15-YYYYMMDD-HHMMSS or v15-<short-sha>

aws ssm send-command \
  --instance-ids "i-026b7477d27bd46d6" \
  --document-name "AWS-RunShellScript" \
  --parameters 'commands=[
    "docker service update --image samkarazi/bebang-erpnext-hrms:v15-PREVIOUS_TAG frappe_backend",
    "docker service update --image samkarazi/bebang-erpnext-hrms:v15-PREVIOUS_TAG frappe_frontend",
    "docker service update --image samkarazi/bebang-erpnext-hrms:v15-PREVIOUS_TAG frappe_websocket",
    "docker service update --image samkarazi/bebang-erpnext-hrms:v15-PREVIOUS_TAG frappe_queue-short",
    "docker service update --image samkarazi/bebang-erpnext-hrms:v15-PREVIOUS_TAG frappe_queue-long",
    "docker service update --image samkarazi/bebang-erpnext-hrms:v15-PREVIOUS_TAG frappe_scheduler"
  ]' \
  --output json

Option C: Return to Docker Compose (Last Resort)

If Swarm has issues, revert to Docker Compose:

aws ssm send-command \
  --instance-ids "i-026b7477d27bd46d6" \
  --document-name "AWS-RunShellScript" \
  --parameters 'commands=[
    "docker stack rm frappe",
    "sleep 30",
    "cd /home/ubuntu/frappe_docker",
    "docker compose -f pwd.yml -f volumes-override.yml up -d"
  ]' \
  --output json

GitHub Secrets Required

These must be set in GitHub repo settings:

DOCKERHUB_USERNAME

DOCKERHUB_TOKEN

AWS_ACCESS_KEY_ID

doppler secrets get AWS_ACCESS_KEY_ID --project bei-erp --config dev --plain

AWS_SECRET_ACCESS_KEY

doppler secrets get AWS_SECRET_ACCESS_KEY --project bei-erp --config dev --plain

Common Issues

Docker Cache Not Picking Up Code Changes (CRITICAL!)

Symptom: Build completes quickly (~2 minutes instead of ~10-15), but new APIs/pages return "module has no attribute" or 404 errors.

Cause: Docker's build cache doesn't detect when git repo contents change during

bench get-app

Build Time Indicator:

no_cache=true

Fix: ALWAYS use

no_cache=true

gh workflow run "Build and Deploy Frappe HRMS" --repo Bebang-Enterprise-Inc/hrms -f no_cache=true -f run_migrate=true

This is NORMAL Docker behavior, not corruption. The

apps.json

Prevention: Use

no_cache=true

• New API files
• Modified API files
• New www pages (like custom login)
• DocType changes
• Any Python code changes

When

no_cache

• Config-only changes (site_config.json)
• Data imports via SQL
• Clear cache operations

Custom Login Page Not Serving (404)

Symptom: Created

hrms/www/login.html

/login

Cause: Frappe's

/login

page_renderer

Fix: Use

website_redirects

# hrms/hooks.py
website_redirects = [
    {"source": "/login", "target": "/bei-login", "redirect_http_status": 302},
]

Then create

hrms/www/bei-login.html

hrms/www/bei-login.py

Verification:

curl -sI https://hq.bebang.ph/login | grep -i location
# Should show: Location: /bei-login

"not whitelisted" Error

Cause: Function not in

@frappe.whitelist()

__init__.py

Fix:

1. Add

   @frappe.whitelist()

   decorator to function
2. Add import to

   hrms/api/__init__.py

3. Rebuild and deploy

Container starts but API fails

Check:

# View logs
aws ssm send-command \
  --instance-ids "i-026b7477d27bd46d6" \
  --document-name "AWS-RunShellScript" \
  --parameters 'commands=["cd /home/ubuntu/frappe_docker && docker compose -f pwd.yml -f volumes-override.yml logs --tail=100 backend"]' \
  --output json

Build fails: "Could not find branch"

Cause: Branch name in apps.json doesn't exist

Fix: Verify branch exists:

git ls-remote https://github.com/Bebang-Enterprise-Inc/hrms production

Port 8080 already in use

Cause: ADMS receiver nginx container uses port 8080

Fix: Use port 80 for Frappe frontend instead:

# In pwd.yml, change frontend ports from 8080:8080 to 80:8080
sed -i "s/8080:8080/80:8080/g" pwd.yml

Frontend container won't start

Cause: Old containers from previous stacks still running

Check:

docker ps -a | grep -E "bebang|frappe"

Fix:

# Stop old stack containers
docker stop bebang-hrms-frontend-1 bebang-hrms-backend-1 ...
docker rm bebang-hrms-frontend-1 bebang-hrms-backend-1 ...

New Domain Returns 404 (Site Not Found)

Symptom: Added new domain (e.g., hq.bebang.ph) but all pages return 404 or "Site not found".

Cause 1: Missing site symlink in

/home/frappe/frappe-bench/sites/

Fix:

# Create symlink for new domain pointing to actual site
aws ssm send-command \
  --instance-ids "i-026b7477d27bd46d6" \
  --document-name "AWS-RunShellScript" \
  --parameters 'commands=["docker exec frappe_docker-backend-1 bash -c \"cd /home/frappe/frappe-bench/sites && ln -sf hrms.bebang.ph hq.bebang.ph\""]'

Cause 2: Wrong

FRAPPE_SITE_NAME_HEADER

FRAPPE_SITE_NAME_HEADER: frontend

FRAPPE_SITE_NAME_HEADER: $host

Fix:

aws ssm send-command \
  --instance-ids "i-026b7477d27bd46d6" \
  --document-name "AWS-RunShellScript" \
  --parameters 'commands=["cd /home/ubuntu/frappe_docker && sed -i \"s/FRAPPE_SITE_NAME_HEADER: frontend/FRAPPE_SITE_NAME_HEADER: \\$host/g\" pwd.yml && docker compose -f pwd.yml -f volumes-override.yml restart frontend"]'

Verify site routing:

curl -sI https://hq.bebang.ph/api/method/frappe.ping | grep -i x-frappe-site-name
# Should show: X-Frappe-Site-Name: hrms.bebang.ph

DO NOT DO

1. NEVER use

   docker commit

   - Corrupts the Python environment (learned 2026-01-22)
2. NEVER edit files directly in container - Changes lost on restart (--preload)
3. NEVER use

   -v

   flag with

   docker compose down

   - Deletes all data volumes
4. NEVER skip

   bench migrate

   after DocType changes
5. NEVER assume server paths - Always verify with

   ls

   first
6. NEVER modify pwd.yml directly without restoring from git first - sed errors accumulate
7. NEVER set

   FRAPPE_SITE_NAME_HEADER: frontend

   - Breaks multi-site routing (learned 2026-01-28)
8. NEVER forget site symlinks for new domains - Each domain needs symlink in sites/ directory (learned 2026-01-28)
9. NEVER run

   bench build

   in production containers - Causes CSS 404 errors (learned 2026-01-28)
10. NEVER run

    bench get-app

    in production containers - Official Frappe Docker policy
11. NEVER use

    docker compose up --no-deps

    - Doesn't refresh assets, causes CSS 404 (learned 2026-01-29)

CRITICAL: Setup Wizard Redirect Loop Issue (2026-01-29) ✅

Status: FIXED (Database Value Changed)

If

/app

setup_wizard.load_languages

Root Cause:

tabDefaultValue

defkey='desktop:home_page' defvalue='setup-wizard'

Fix:

-- Run via bench console or direct MariaDB
UPDATE tabDefaultValue SET defvalue='Workspaces'
WHERE defkey='desktop:home_page' AND parent='__default';

Also ensure these are set:

-- Mark setup as complete
UPDATE tabDefaultValue SET defvalue='1'
WHERE defkey='setup_complete' AND parent='__default';

-- Mark HRMS app as setup complete
UPDATE `tabInstalled Application` SET is_setup_complete=1
WHERE app_name='hrms';

Clear cache after:

docker compose -f pwd.yml -f volumes-override.yml exec -T backend bench --site hrms.bebang.ph clear-cache

Reference:

docs/plans/FRAPPE_DEPLOYMENT_FIX_PLAN_2026-01-29.md

CSS 404 Asset Desync Issue - RESOLVED (2026-01-29, Updated 2026-01-29) ✅

Status: PERMANENTLY FIXED

The GitHub Actions workflow has been updated to prevent CSS 404 errors. Every deployment now:

1. Runs Swarm rolling updates (recreates containers with fresh assets)
2. Flushes Redis cache (prevents stale HTML serving old asset hashes)
3. Verifies CSS/JS assets load correctly after deploy (fails workflow on 404)

Commits:

• 48f1c2f9b

  - fix: Add asset volume reset and CSS health check to deployment

• TBD

  - fix: Add Redis flush step to prevent stale HTML caching

The Original Problem

Running

bench build

bench migrate

• Browser requests

  desk.bundle.ABC123.css

• Server only has

  desk.bundle.XYZ789.css

• Result: Entire application breaks (no styling)

This caused 4 production outages, now prevented by workflow fix.

Root Cause

From GitHub Issue #790:

"Any new nginx images which have new assets at

/usr/share/nginx/html

, will not propagate these changes if the volume already exists."

Architecture flaw:

1. Assets are baked into Docker image at build time
2. Assets volume (

   bebang-hrms_sites

   ) persists between deployments

3. bench migrate

   regenerates assets in backend container
4. Frontend container still has OLD assets from image/volume

5. assets.json

   points to NEW hashes, CSS files have OLD hashes

Official Frappe Policy

From Frappe Docker FAQ:

"You cannot build assets using

bench build

in running production containers. It will mess up the attached assets volume."

The Fix: Delete Assets Before Deploy

ALWAYS delete the assets volume before deployment:

# Add to GitHub Actions BEFORE docker compose up
docker compose -f pwd.yml -f volumes-override.yml down
docker volume rm bebang-hrms_assets 2>/dev/null || true
docker compose -f pwd.yml -f volumes-override.yml up -d

Or via SSM:

aws ssm send-command \
  --instance-ids "i-026b7477d27bd46d6" \
  --document-name "AWS-RunShellScript" \
  --parameters 'commands=["cd /home/ubuntu/frappe_docker && docker compose -f pwd.yml -f volumes-override.yml down && docker volume rm bebang-hrms_assets 2>/dev/null || true && docker compose -f pwd.yml -f volumes-override.yml up -d"]'

Permanent Fix: Docker Swarm - NOW ACTIVE ✅

Docker Swarm was deployed on 2026-01-29. It automatically recreates containers on deployment:

# Swarm is already initialized and running
docker service ls  # View all 9 services

# Deploy updates via rolling updates (zero downtime)
docker service update --image samkarazi/bebang-erpnext-hrms:v15-newtag frappe_backend

Why Swarm works: Rolling updates recreate containers with fresh assets, eliminating stale CSS/JS issues.

Volume Safety Reference

bebang-hrms_sites

bebang-hrms_db-data

bebang-hrms_logs

bebang-hrms_assets

Your DocTypes, employees, APIs - all safe. Only compiled CSS/JS is deleted and recreated.

EMERGENCY FIX: CSS 404 (If It Happens Again)

Fastest fix (30 seconds):

export AWS_ACCESS_KEY_ID=$(doppler secrets get AWS_ACCESS_KEY_ID --project bei-erp --config dev --plain)
export AWS_SECRET_ACCESS_KEY=$(doppler secrets get AWS_SECRET_ACCESS_KEY --project bei-erp --config dev --plain)
export AWS_DEFAULT_REGION=ap-southeast-1

# 1. Force redeploy all services (recreates containers with fresh assets)
aws ssm send-command --instance-ids "i-026b7477d27bd46d6" --document-name "AWS-RunShellScript" \
  --parameters 'commands=[
    "docker service update --force frappe_backend",
    "docker service update --force frappe_frontend"
  ]'

# 2. Wait 60 seconds for services to converge

# 3. Flush Redis (CRITICAL - clears cached HTML with old hashes)
aws ssm send-command --instance-ids "i-026b7477d27bd46d6" --document-name "AWS-RunShellScript" \
  --parameters 'commands=[
    "docker exec $(docker ps -qf name=frappe_redis-cache) redis-cli FLUSHALL",
    "docker exec $(docker ps -qf name=frappe_redis-queue) redis-cli FLUSHALL"
  ]'

Why this works: The CSS 404 happens when:

1. assets.json

   has hash A (from cache/migrate)
2. Actual CSS files have hash B (from Docker image)
3. Redis caches HTML referencing hash A
4. Force redeploy syncs assets.json to hash B
5. Redis flush clears cached HTML, forcing fresh generation with hash B

How to Diagnose CSS 404

# 1. Check what HTML is requesting
curl -s https://hq.bebang.ph/ | grep -oP "website\.bundle\.[A-Z0-9]+\.css"

# 2. Check what CSS files actually exist (Swarm)
aws ssm send-command --instance-ids "i-026b7477d27bd46d6" --document-name "AWS-RunShellScript" \
  --parameters 'commands=["docker exec $(docker ps -qf name=frappe_frontend) ls /home/frappe/frappe-bench/sites/assets/frappe/dist/css/ | grep website"]'

# 3. Check what assets.json says
aws ssm send-command --instance-ids "i-026b7477d27bd46d6" --document-name "AWS-RunShellScript" \
  --parameters 'commands=["docker exec $(docker ps -qf name=frappe_backend) cat /home/frappe/frappe-bench/sites/assets/assets.json | grep website.bundle.css"]'

# 4. If hashes don't match → Force redeploy + Redis flush (see above)

Research Reference

Full research report:

docs/reports/FRAPPE_DOCKER_DEPLOYMENT_RESEARCH_2026-01-28.md

Sources:

• GitHub Issue #790
• Frappe Docker FAQ
• Community Discussion

Server Discovery Commands

If you're unsure about server configuration, run these:

# List home directory to find frappe location
aws ssm send-command --instance-ids "i-026b7477d27bd46d6" --document-name "AWS-RunShellScript" \
  --parameters 'commands=["ls -la /home/ubuntu/"]' --output text --query 'Command.CommandId'

# List docker volumes
aws ssm send-command --instance-ids "i-026b7477d27bd46d6" --document-name "AWS-RunShellScript" \
  --parameters 'commands=["docker volume ls | grep -E bebang|frappe|sites"]' --output text --query 'Command.CommandId'

# List running containers
aws ssm send-command --instance-ids "i-026b7477d27bd46d6" --document-name "AWS-RunShellScript" \
  --parameters 'commands=["docker ps --format \"{{.Names}} {{.Image}}\""]' --output text --query 'Command.CommandId'

# Check what's using a port
aws ssm send-command --instance-ids "i-026b7477d27bd46d6" --document-name "AWS-RunShellScript" \
  --parameters 'commands=["lsof -i :8080 || ss -tlnp | grep 8080"]' --output text --query 'Command.CommandId'

# Get command output
aws ssm get-command-invocation --command-id "<COMMAND_ID>" --instance-id "i-026b7477d27bd46d6" \
  --query '[Status, StandardOutputContent]' --output text

Files Reference

.github/helper/apps.json

.github/workflows/build-and-deploy.yml

hrms/api/__init__.py

hrms/api/*.py

Related Skills

• /local-frappe

  - Test locally first before production deployment

• /frappe-sql-bulk

  - For data imports (no rebuild needed)

• /frappe-external-app

  - For React apps via API