The Namada Dry Run was designed as a high-fidelity simulation to test the robustness of the protocol, governance processes, and community involvement before fully operational deployment to mainnet. By systematically exploring real-world scenarios in a controlled environment, the Namada team aimed to refine their systems, identify vulnerabilities, and ensure network stability. This was done by validating key components such as IBC interactions and rate limits with new governance processes. 

Phase 1: Initial Setup

• Objective: Establish the testnet, distribute tokens, and engage participants.
• Key Activities:
  • Creation and distribution of genesis files.
  • Coordination with participants for testnet launch.
  • Deployment of initial validators and basic features.

Lessons Learned:

• Initial Coordination: Community-driven engagement proved effective, but clearer guidelines and resource access were necessary.
• Technical Challenges: Some validators encountered setup issues due to unoptimized documentation.

Phase 2: Governance Process Testing

• Objective: Test the on-chain governance mechanism.
• Key Activities:
  • Participants proposed and voted on governance decisions.
  • Real-world voting scenarios simulated.
  • Proposal related to IBC rate adjustments evaluated.

Lessons Learned:

• Community Dynamics: High participation highlighted the enthusiasm, but diverse viewpoints underscored the need for more structured debates.
• Tool Improvements: Governance interface enhancements were identified to streamline future proposals.

Accomplishments:

• Successful deployment of governance proposals with significant voter turnout.
• Strengthened understanding of proposal lifecycle and feedback mechanisms.

Phase 3: Expanding IBC Capabilities

• Objective: Test and validate inter-blockchain communication (IBC) features, especially with USDC Noble Mainnet.
• Key Activities:
  • Increased IBC rate limits to test cross-chain functionality.
  • Observed how transactions scaled with higher usage.

Lessons Learned:

• Rate Adjustments: Proper rate-limit configuration was crucial to avoid congestion without over-provisioning resources.
• Cross-chain Resilience: Minor inconsistencies in transaction propagation were addressed promptly.

Accomplishments:

• Smooth IBC transaction processing between Namada and partner chains.
• Key insights into optimizing rate limits for mainnet scaling.

Phase 4: Stress Testing Network Performance

• Objective: Simulate high-traffic scenarios to measure system robustness.
• Key Activities:
  • Orchestrated high-volume transactions to test validator capacity.
  • Monitored latency, block times, and network throughput.

Lessons Learned:

• Bottlenecks: Identified specific nodes with delayed updates during peak loads.
• Optimization Needs: Required more advanced monitoring tools to better analyze stress data.

Accomplishments:

• Validator resilience under stress.
• Improved protocol adjustments to reduce latency in high-traffic conditions.

Phase 5: Consolidation and Final Adjustments

• Objective: Review cumulative findings and finalize the protocol for mainnet readiness.
• Key Activities:
  • Comprehensive feedback sessions with participants.
  • Iterative protocol updates based on insights from earlier phases.

Lessons Learned:

• Documentation and Support: Improved technical documentation was pivotal for successful community participation.

Accomplishments:

• Network parameters finalized for mainnet readiness.
• A stronger, more cohesive community aligned for the launch.

Challenges and Solutions from Namada's Dry Run 

Short Notice and Timing: The upgrade notice was short and not widely disseminated, coinciding with the US Thanksgiving holiday 

Solution: Future upgrades will be coordinated well in advance on weekdays that are not major holidays. Two governance proposals will be used: one to decide on the upgrade and another to decide on the date and time. The validator alerts mailing list, Discord channel, and readiness confirmations from non-voting validators will also be utilized 

Dominant Validator and Incorrect Signing: A validator increased their stake, gaining 18.75% of the voting power, necessitating more voting power from other validators to proceed. The lack of correctly-signing voting power also highlighted issues with certain validator operators.

Solution: The mainnet launch is expected to attract a substantial increase in stake, leading to a better distribution of voting power. Additionally, the importance of careful verification and adherence to guidelines was emphasized to validator operators. This includes verifying the SHA256 sum of the migration JSON file, pre-verifying scripts, and verifying and potentially renaming replaced binaries to avoid accidental usage.

Operator Struggles: The upgrade process exposed struggles faced by some validator operators. Some validators restarted their nodes after reaching the halt block height, leading to errors. Some validators encountered setup issues due to unoptimized documentation.

Solution: This highlighted the importance of following the upgrade guide meticulously. Allowing the node to stop as instructed and setting the service template to "Restart=False" before proceeding with the upgrade can prevent this issue.

Slow Recovery: The 211GB state size made creating and distributing snapshots of the pre-upgrade chain time-consuming. The client also corrupted the database upon restart.

Solution: Namada v0.46.0 addresses the state size issue. The migration process and code have been improved to prevent database corruption. Creating a snapshot of the pre-upgrade chain is recommended as a precaution.

Despite these challenges, the dry run was considered a success, achieving approximately 84% online voting power on proposals. The event allowed for valuable learning and improvements, preparing Namada for any state migrations and more required post-mainnet launch.