The Security Side Effect: How Network Automation Accidentally Transformed Our Risk Posture

Summarizing Lee Harper's practical automation journey from AutoCon3

"We didn't go into this thinking of automation as a security tool," admitted Lee Harper, Enterprise Administrator at Terricon Consultants. "We went into this trying to free up resources without spending money on potentially an employee that would or would not work out."

What happened next was an unexpected transformation that took their organization from 34 security findings to just 9 in two years—all while growing 50% without adding network team members.

The Acquisition-Driven Challenge

Terricon, an engineering consulting firm with locations throughout the continental United States, grew primarily through mergers and acquisitions. This growth strategy created a unique automation driver: "When you grow through acquisitions, you grow in jumps. We'd acquire a new company and suddenly have five new locations to onboard."

By 2019, they managed over 120 locations with a traditional stack (router, firewall, WAN optimizer, switches) connected via MPLS to their Kansas City data center. Their four-person network team was "pretty much fully loaded" with no extra time for anything else.

Without desire to add headcount, they needed a strategy that would let them onboard automation with existing resources.

The Snowball Strategy

Harper's team adopted what he called "the snowball theory"—similar to paying off credit cards by starting with smallest balances first. They focused on critical projects and low-hanging fruit that could be implemented quickly and produce immediate results.

"As you implement these low-hanging fruit, the resources you were putting into doing these projects are freed up, and then you have those resources to implement the next more demanding aspect of your automation. Like a snowball rolling downhill, your project grows and your resources grow."

The Implementation Journey

Phase 1: Inventory Management Starting with Gluware's basic inventory module took about a month, mostly spent adjusting ACLs and credentials so the platform could access equipment.

Phase 2: Static Configuration They began standardizing basic configurations—DNS servers, NTP settings—things that didn't change but needed to be consistent across platforms.

Phase 3: OS Standardization A critical realization emerged: they had countless different OS versions across the same hardware platforms. Each new device was shipped with the then-current recommended version and never updated until retirement.

"We had problems with syntax changes between versions, features implemented later that older versions didn't support." Standardizing on single OS versions took 3-4 months, mostly spent testing upgrade procedures from all different versions.

Phase 4: Advanced Configuration Management With standardized OS versions, they could return to configuration management with confidence. Variable management became crucial for handling site differences.

The Results: Beyond Time Savings

The immediate operational improvements were dramatic:

  • Complete network inventory with firmware and hardware versions tracked

  • Standardized OS versions across all platforms

  • Configuration changes reduced from 3-4 days (entire team) to 3-4 hours (single person)

But the security benefits emerged as an unexpected bonus.

The Security Transformation

Configuration Drift Detection: Validating configuration changes became automatic, catching fat-finger errors and documenting OS update impacts. This provided "upper management a lot of peace with OS updates" due to increased safety.

Third-party managed systems like SD-WAN presented particular challenges: "Updates weren't well documented. We'd push an update and suddenly have a bunch of broken offices." Configuration drift detection in lab environments first prevented "potentially catastrophic effects of undocumented changes."

Configuration Auditing: Automated checks against security benchmarks like CIS (Center for Internet Security) standards ensured continuous compliance. As Harper noted, "Most tools we looked at maybe half of the benchmarks were automated, everything else was manual, and the time investment was just not appetizing."

Live State Auditing: Real-time device connection allowed checking hardware status and service availability. When new vulnerabilities emerged, they could quickly verify if their configurations were even affected: "Being able to write a real quick check to run against all affected hardware—does our configuration even affect this vulnerability?"

The Security Workflow

The automation created a three-part security approach:

Discovery: Complete inventory and configuration auditing revealed which vulnerabilities actually affected their environment, "saving you a lot of time chasing down vulnerabilities that don't even affect you."

Remediation: Configuration templates enabled rapid response to unauthorized changes and new vulnerabilities. "When a new vulnerability comes out and you need to adjust your config, you just go into your template, make the necessary change, push it all out again."

Reporting: Automated documentation proved compliance efforts. "If it's not documented, you aren't doing it. Being able to report on all your vulnerability remediation, all your benchmarks—being able to put out a report that says yes, we are doing it, yes we are compliant."

Measurable Security Improvements

The transformation delivered quantifiable results:

  • Growth: 50% expansion (120 to 200+ locations) without adding team members

  • Infrastructure overhaul: Complete migration from MPLS to SD-WAN and switch platform replacement

  • Security audit results: Reduced from 34 findings to 9 with their cybersecurity insurer

  • Response time: Firmware rollouts for vulnerability patches completed in single nights

  • Business impact: Won engineering contracts specifically due to improved security posture

"Right now we're one of their favorite customers because we're not a risk for them," Harper said of their cybersecurity insurer.

Phase Three: Integration and Automation

Looking forward, Terricon is focusing on API integrations with third-party platforms (SD-WAN, Palo Alto firewalls via Panorama) and ticketing systems for automated response.

Workflow automation remains challenging because "there's still a lot of manual processes involved in kicking off the automated aspects" due to changes requiring human validation.

Their goals include:

  • Central point of change to eliminate human error

  • Faster incident response through automated ticketing workflows

  • Better team communication through automated change notifications

  • Integration with cloud systems (like automatically updating Azure ACLs with office public IPs)

The Philosophy Debate

When questioned about his "implement first, design later" approach versus Claudia de Luna's design-first keynote philosophy, Harper offered a pragmatic response:

"The biggest issue is if you're constantly designing and never implementing, what's the point? You have to actually put things into effect for automation to work, and that's part of selling it to upper management."

He acknowledged potential refactoring needs but emphasized choosing adaptable tools: "With a good tool that shouldn't be a huge issue because you're laying groundwork for being able to implement your system."

The Unexpected Security Champion

Harper's story illustrates how network automation can become a powerful security tool even when that wasn't the original intent. By focusing on operational efficiency and standardization, they accidentally built a robust security posture that:

  • Provides complete visibility into network state

  • Enables rapid response to vulnerabilities

  • Ensures continuous compliance with security benchmarks

  • Documents all changes for audit purposes

  • Reduces human error through standardization

The lesson? Sometimes the best security improvements come not from security-focused projects, but from operational excellence initiatives that inherently improve risk posture.

As Harper concluded, "Automation just made security easier." For organizations struggling to justify automation investments, the security benefits might provide the business case that pure operational efficiency cannot.


Chris Grundemann

Executive advisor. Specializing in network infrastructure strategy and how to leverage your network to the greatest possible business advantage through technological and cultural transformation.

https://www.khadgaconsulting.com/
Previous
Previous

Beyond Building Networks: The Day Two Automation Challenge

Next
Next

The Network Automation Epic: Terraform's Journey from Cloud to Campus