From Scripts to Microservices: The Evolution of Enterprise Network Automation
Summarizing Shirish Basant Rai's architectural journey from AutoCon3
"Just do it," Shirish Basant Rai concluded his talk with a deceptively simple message. "Don't ask questions, don't debate too much... try it. And if it works, maybe there's something else that does it better. But you can change it any time."
But the journey he described—transforming network automation from individual Python scripts to a full microservices architecture—was anything but simple. It's a roadmap for organizations ready to take their automation efforts from hobby projects to enterprise-grade platforms.
The Familiar Starting Point
Rai painted a picture many network teams will recognize: engineers scattered across the organization, each maintaining their own Python scripts and Ansible playbooks on their laptops. It worked, but it didn't scale. Manual intervention was still required, code diverged across teams, and the risk of network outages remained high.
The first evolution was predictable—centralize everything into Git-based platforms and standardize on Ansible playbooks for service provisioning. "They were successful, they were doing the job," Rai noted. "But very quickly you hit a wall."
When Orchestration Isn't Enough
The team's next step involved implementing a workflow orchestrator with NSO (Network Services Orchestrator) providing abstraction across multiple domains—optical, SD-WAN, IT, NFV, and packet networks. For a moment, they thought they were done.
"We thought, oh, we've done the job. It's done now. We don't need to touch it again. Just keep developing workflows and NSO YANG models and templates."
Reality had other plans. As use cases became more complex, critical gaps emerged:
The Async Problem: Workflow engines don't provide RESTful interfaces. You send a request, it does something asynchronously, and you have to listen for events to know what happened.
Cross-Domain Complexity: Activating a customer premises equipment first, then an E-line service a week later, followed by IP VPN service an hour after that—this sequential, time-delayed orchestration was nearly impossible with traditional workflow approaches.
Conflict Management: Multiple activation systems hitting the same workflows could conflict with each other, with no clear way to manage concurrent requests.
The Microservices Evolution
Rai's team turned to software engineering best practices, specifically microservices architecture. "It felt different, like 'oh, that's really complex,'" he admitted. "But the beginning is hard. We just need to start."
The principles were straightforward:
Single responsibility: Each microservice does exactly one thing
Self-contained: Own storage, own database, no shared dependencies
Independent scaling: Scale individual services based on demand
Easy deployment: Template-based approach for rapid development
The Mature Architecture
The final architecture reveals sophisticated thinking about enterprise automation:
API Gateway: A single entry point that abstracts the internal complexity from integration teams. "Keep it simple," Rai emphasized. "They don't need to know anything below that."
Service Manager: Provides the RESTful APIs that were missing from workflow engines, maintaining request and service lifecycles with persistent state management.
Specialized Microservices: Complex logic gets its own dedicated services—IP allocation, BGP filter management, authentication systems, AI agents.
Orchestration Layer: Communicates between components, abstracts integrations, and packages payloads for domain-specific controllers.
Practical Benefits in Action
The results speak for themselves:
Independent development and deployment: Fix a bug in IP allocation without touching anything else
Targeted scaling: Handle API floods on specific services without over-provisioning everything
Conflict prevention: The service manager polices concurrent requests, declining or queuing conflicting operations
Resilience: When Kafka fails, the microservices architecture provides fallback for event-driven processes
One telling example: their IP allocation system now handles 100,000 API calls per second. "When they start seeing the benefit," Rai observed, "then they're like aligning slowly into, 'oh, okay, I'm getting it now.'"
The Implementation Reality
When asked about team size, Rai's answer was surprisingly modest: "We partnered with Ductus and with a team of four to eight engineers, they can very easily pick it up and build that slowly and scale it out."
The key insight? You don't build it all at once. Start with workflows, understand the architecture, build the CI/CD processes around it, then evolve to the next level.
The People Challenge
Technical architecture was only half the battle. "The biggest challenge," Rai admitted, "is people." Even after deploying to production, he was still explaining the service manager concept to team members.
His advice: "Bring them on board very early on. Bring the operational team very early on... They will not understand the architectural vision initially, but once they see the benefit... they start aligning."
Lessons for the Journey
Start Simple: Begin with a new use case, not an existing one. It's easier to establish patterns with a clean slate.
Define Principles Early: Draw your architecture and understand the direction, even if it changes completely in a month.
Fail Fast: Don't get paralyzed by tool selection. Try something quickly, learn from failures, iterate.
Document Everything: Clear documentation, versioning, and release strategies become critical as complexity grows.
Think Security Later: Get the architecture working first, then secure it before production. Don't let security concerns prevent experimentation.
The Bigger Picture
What makes this presentation valuable isn't just the technical architecture—it's the honest portrayal of evolution. Most organizations won't jump directly from Ansible playbooks to microservices. They'll follow a similar progression: scripts → centralized automation → orchestration → microservices.
Each step solves real problems while revealing new challenges. The key is recognizing when you've outgrown your current approach and being willing to evolve.
As Rai concluded, "Build a good foundation. It will feel complex, but it gets easier... When you get more and more use cases, you create your architecture and new use cases become easier to implement."
Sometimes the path to simplicity runs through complexity. The question isn't whether to make the journey, but when you're ready to take the next step.
Watch the full presentation: Taking Automation to Next Level: Drive Your Network with Microservices
