Debug Hyperlane Message Skill

When to Use

• User provides a message ID (e.g.,

  0xa454...

  ) or explorer URL
• Investigating why a message is stuck or not delivered
• Debugging gas estimation failures
• Understanding message processing status

Input Parameters

message_id

0xa454559c...

explorer_url

https://explorer.hyperlane.xyz/message/0x...

Debugging Workflow

Step 1: Get Basic Message Details from Explorer

Fetch the explorer page to get origin/destination chains and basic status:

WebFetch: https://explorer.hyperlane.xyz/message/[MESSAGE_ID]
Prompt: Extract message status, origin chain, destination chain, sender, recipient, timestamp, delivery status

Key info to extract:

• Origin chain and domain ID
• Destination chain and domain ID
• Is it delivered? (if yes, no debugging needed)
• Timestamp (how old is it?)

Step 2: Search Relayer Logs for the Message

Use the gcloud CLI to find logs related to this message in the omniscient relayer:

gcloud logging read 'resource.type="k8s_container" AND resource.labels.project_id="abacus-labs-dev" AND resource.labels.location="us-east1-c" AND resource.labels.cluster_name="hyperlane-mainnet" AND resource.labels.namespace_name="mainnet3" AND labels.k8s-pod/app_kubernetes_io/component="relayer" AND labels.k8s-pod/app_kubernetes_io/instance="omniscient-relayer" AND labels.k8s-pod/app_kubernetes_io/name="hyperlane-agent" AND "[MESSAGE_ID]"' --project=abacus-labs-dev --limit=50 --format=json --freshness=1d

Step 3: Identify the Message Status

Look for the message in

PendingMessage

Retry(ErrorEstimatingGas)

Retry(GasPaymentRequirementNotMet)

Retry(CouldNotFetchMetadata)

FirstPrepareAttempt

Extract status with:

grep -o "message_id: [MESSAGE_ID][^}]*" [log_output] | sort -u

Step 4: Check for Gas Payment Issues

If message is slow/stuck, search for gas payment evaluation logs:

gcloud logging read '[BASE_RELAYER_QUERY] AND "[MESSAGE_ID]" AND jsonPayload.fields.message:"Evaluating if message meets gas payment requirement"' --project=abacus-labs-dev --limit=5 --format=json --freshness=7d

Key fields in

jsonPayload.fields

• current_payment.gas_amount

  - gas units paid for by sender

• tx_cost_estimate.gas_limit

  - gas units needed for delivery

• current_expenditure.gas_used

  - gas already spent on retries

• policy

  - subsidy policy (e.g.,

  fractional_numerator: 1, fractional_denominator: 2

  = 50% subsidy)

If

gas_amount < gas_limit

"Repreparing message: Gas payment requirement not met"

Step 5: For Gas Estimation Errors - Get the Revert Reason

If status is

ErrorEstimatingGas

jsonPayload.fields.error

# Query logs with error field
gcloud logging read 'resource.type="k8s_container" AND resource.labels.project_id="abacus-labs-dev" AND resource.labels.location="us-east1-c" AND resource.labels.cluster_name="hyperlane-mainnet" AND resource.labels.namespace_name="mainnet3" AND labels.k8s-pod/app_kubernetes_io/component="relayer" AND labels.k8s-pod/app_kubernetes_io/instance="omniscient-relayer" AND labels.k8s-pod/app_kubernetes_io/name="hyperlane-agent" AND "[MESSAGE_ID]" AND jsonPayload.fields.error:*' --project=abacus-labs-dev --limit=5 --format=json --freshness=1d 2>/dev/null | grep -o '"error": "[^"]*"' | head -1

The error field contains the full revert reason, e.g.:

"error": "ContractError(...JsonRpcError { code: 3, message: \"execution reverted: panic: arithmetic underflow or overflow (0x11)\", data: Some(...) }...)"

Quick extraction - pipe to extract just the revert message:

... | grep -oP 'execution reverted: [^"\\]+' | head -1

Common revert patterns:

• execution reverted: panic: arithmetic underflow or overflow (0x11)

  - Contract math error

• execution reverted: [CUSTOM_ERROR]

  - Custom contract revert (decode with

  cast 4byte

  )

• execution reverted

  with hex data - Decode selector with

  cast 4byte 0x[first4bytes]

Step 6: Extract Message Details

From the logs, identify:

• origin

  : Source chain

• destination

  : Destination chain (or domain ID like

  4114

  for Citrea)

• sender

  : Origin contract address

• recipient

  : Destination contract address (the warp route or recipient)

• nonce

  : Message sequence number

Example log format:

HyperlaneMessage { id: 0x..., nonce: 162898, origin: ethereum, sender: 0x..., destination: 4114, recipient: 0x... }

Step 7: Report Findings

Summarize:

1. Message ID: Full ID
2. Route: Origin -> Destination (e.g., Ethereum -> Citrea)
3. Status: Current processing status
4. Root Cause: The actual error (e.g., contract revert reason)
5. Retry Count: How many times it's been attempted
6. Recommendation: What needs to happen (e.g., fix recipient contract, wait for validators)

Common Root Causes

Gas Estimation Errors

panic: arithmetic underflow or overflow (0x11)

IXERC20_NotHighEnoughLimits()

InsufficientBalance

Unauthorized

Metadata/Validator Issues

If

CouldNotFetchMetadata

1. Check validator status using the

   debug-validator-checkpoint-inconsistency

   skill
2. Identify if validators are behind on the origin chain

Gas Payment Issues

If

GasPaymentRequirementNotMet

• Compare

  current_payment.gas_amount

  vs

  tx_cost_estimate.gas_limit

• Message will auto-retry every ~3 min until gas prices drop or subsidy kicks in
• Check

  policy

  field for subsidy ratio (e.g., 1/2 = relayer covers 50%)
• Resolution: wait for gas prices to drop, or manually subsidize via IGP top-up

Decoding Revert Selectors

When you see hex revert data like

0x4e487b71...

cast 4byte 0x4e487b71
# Returns: Panic(uint256)

Common panic codes:

• 0x11

  - Arithmetic overflow/underflow

• 0x12

  - Division by zero

• 0x21

  - Invalid enum value

• 0x31

  - Pop on empty array

• 0x32

  - Array out of bounds

Domain ID Reference

Common domain IDs (destination field in logs):

• 1

  - Ethereum

• 42161

  - Arbitrum

• 10

  - Optimism

• 137

  - Polygon

• 4114

  - Citrea

Check

@hyperlane-xyz/registry

Example Investigation

User asks: "Why isn't message 0xa454... being processed?"

1. Fetch explorer: Origin=Ethereum, Dest=Citrea, not delivered
2. Query relayer logs for

   0xa454...

3. Find status:

   Retry(ErrorEstimatingGas)

4. Search error logs:

   execution reverted: panic: arithmetic underflow or overflow (0x11)

5. Report: "Message from Ethereum to Citrea is failing because the recipient contract (0xbd39...) on Citrea is reverting with an arithmetic overflow error. This is a bug in the destination contract that needs to be fixed by the contract owner."