Foundation, Foundation, Foundation

Imagine spending the better part of an afternoon painting a fence.

Stroke.

Dip.

Stroke.

Dip.

A few hours go by and you're finally starting to see progress.

Then you look back and realize the neighbor kid has been throwing clods of dirt at the part you already finished.

That is sometimes what trying to introduce programmability and automation into networking feels like.

You can have the scripts.

You can have the APIs.

You can have the Python libraries, the templates, the source of truth, the automation servers, and the best intentions in the world.

But if the foundation is not there, one person can undo a surprising amount of work.

Sometimes intentionally.

Usually unintentionally.

Either way, the result is the same.

You spend your time repainting the fence.

Everyone Has to Row in the Same Direction

Sometimes the crew isn't just rowing in different directions. Sometimes one guy is chewing on the oar.

When programming concepts are applied to networking, a foundation has to exist first.

Standards.

Processes.

Trust.

Access.

Consistency.

People who understand what the automation is trying to accomplish.

And, perhaps most importantly, support from the people responsible for the environment.

I can already hear somebody saying:

One person can't disrupt an automation effort that badly.

Sure they can.

Imagine building an automation platform that depends on reaching a set of network devices from a known group of systems.

Then somebody modifies an access control list.

Maybe they do not know those systems are automation hosts.

Maybe they do know and have a different idea of how things should work.

Either way, your automation stops.

Nothing about Python changed.

Nothing about your code changed.

Nothing about the API changed.

The environment changed underneath you.

And suddenly the problem is no longer technical.

Technology Can't Solve Every Problem

That has probably been one of the more frustrating lessons of my career.

Technology can solve a lot of things.

It cannot make people cooperate.

It cannot make an organization agree on standards.

It cannot make a manager support a direction they do not believe in.

It cannot make a process consistent when everybody is allowed to invent their own version of it.

It cannot turn a dysfunctional environment into a functional one merely because somebody installed Ansible.

Sometimes the biggest obstacle to automation has absolutely nothing to do with automation.

When Networking and Programming Started Dating

There was a period when networking and programming were just beginning to seriously meet each other.

Today, Python in networking barely raises an eyebrow.

Back then, it was different.

Some of the libraries and tools I wanted to use ran cleanly on Linux and were, at best, unpredictable on Windows.

You could sometimes make them work.

Sometimes.

Other times you spent hours fighting dependencies, compilers, strange Python packaging behavior, or libraries that clearly assumed they were running on a Unix-like system.

Linux was simply the better tool for the job.

The problem was that sometimes management would say:

No Linux.

That could be frustrating.

You could lay out the possibilities.

You could explain how automation might reduce repetitive work.

You could show how consistency could improve.

You could talk about configuration generation, validation, reporting, auditing, backups, testing, and all of the other things that become possible once networking starts borrowing ideas from software development.

You could show the horizon.

But if the answer was no, the answer was no.

Technology does not override organizational authority.

One Person Can Only Push So Far

That is probably the heart of this post.

One person can build a lot.

One person can learn a lot.

One person can prototype.

One person can demonstrate what is possible.

One person can even build something from the ground up.

But one person cannot create organizational alignment by themselves.

That can be incredibly frustrating.

You may look at a decision and think it makes absolutely no sense.

You may see people doing things manually that could have been automated years ago.

You may see inconsistent configurations, undocumented changes, one-off exceptions, and processes that exist mostly because that is how somebody has always done them.

Eventually I learned that there is a difference between recognizing dysfunction and believing you are personally responsible for fixing all of it.

If an opportunity appears to improve something, take it.

If people want help fixing a broken process, help fix it.

But if the people, processes, or organization do not want to change, there is only so much force you can apply before you are just exhausting yourself.

Sometimes People Have Reasons You Cannot See

Years ago I worked for a manager who was actually a pretty sharp guy.

He understood technology.

He understood people.

He understood how to manage.

But when it came to applying programming techniques to networking, we rarely seemed to see things the same way.

At the time, I thought one of two things was happening.

Either he simply did not like me, or he really did not like the direction I was trying to take networking.

I eventually leaned toward the second explanation.

Later, I learned that earlier in his career he had worked for a company that was doing some very advanced work for the time around combining programming and networking.

From what I understood, that experience had not gone particularly well.

I can only speculate, but I have wondered whether that experience shaped how he viewed the same ideas later.

And honestly, that would make sense.

People do not walk into a meeting as blank slates.

Past failures matter.

Past outages matter.

Bad projects matter.

Being burned by a technology once can make somebody extremely cautious about touching it again.

At the time, though, I did not understand that.

All I could see was resistance.

Then Something Changed

Fast forward several years.

Maybe close to a decade.

People move on.

He did too.

He moved upward, eventually became a CIO, and later went on to bigger things.

Eventually he landed at a very large global organization.

And that organization was heavily invested in things that, years earlier, I had been trying to talk about:

Linux.

Automation.

Agile methods.

Programming applied to infrastructure.

Infrastructure being treated more like software.

After he had been there for a few years, our paths crossed again.

And something was different.

The resistance I remembered was gone.

The ideas that had once been difficult to even discuss were now completely normal.

Maybe it was exposure.

Maybe the technology matured.

Maybe the tools got better.

Maybe the surrounding culture changed.

Maybe his earlier concerns were absolutely valid for the environment we were in at the time.

Probably some combination of all of the above.

Whatever the reason, it reminded me of something important:

People change.

Technology changes.

Organizations change.

And sometimes an idea that looks ridiculous in one environment becomes obvious in another.

Don't Stop Learning

That experience is one of the reasons I believe so strongly in continuing to learn even when your current environment does not give you much opportunity to use what you are learning.

If you have to build a little lab at home, build the lab.

If that turns into a mini data center in the garage, well...

Welcome to the club.

Learn Linux.

Learn Python.

Learn APIs.

Learn automation.

Learn how the network actually behaves underneath all of those tools.

Learn because you are curious.

Learn because someday the environment around you may finally catch up.

You never know when the opportunity is going to appear.

Foundation, Foundation, Foundation

The tools matter.

The code matters.

The architecture matters.

But the foundation matters more.

It is very hard to get anywhere when you have a boat full of rowers all rowing in different directions.

I am pretty sure I have even seen one guy start chewing on his oar.

Automation works best when everybody agrees on some basic truths.

Where configuration should come from.

Who is allowed to change it.

How changes are documented.

What the source of truth is.

What standards are expected.

What systems automation depends on.

What happens when reality no longer matches the template.

Without that foundation, automation becomes fragile.

And that leads to another lesson.

Defensive Programming Is Not Paranoia

Over the years, I have become much more defensive in how I write network automation.

Not defensive in the violent sense.

Defensive in the programming sense.

Check everything.

Validate everything.

Sanitize input.

Normalize data.

Verify assumptions.

Expect drift.

Expect somebody to change something.

Expect the network you saw yesterday to be slightly different today.

If an ACL is supposed to be deployed everywhere, do not assume it is.

Check.

If an interface is supposed to have a particular description, do not assume it does.

Check.

If a configuration template says one thing and the device says another, treat that difference as information.

The network is not a static database.

It is a living environment touched by people, processes, maintenance windows, vendors, outages, troubleshooting sessions, emergency fixes, forgotten exceptions, and the occasional mystery command nobody remembers entering.

The safest assumption is that assumptions expire.

That does not mean automation is pointless.

It means good automation has to account for the fact that reality moves.

You Can't Script Culture

That may be the biggest lesson of all.

You can script configuration.

You can script validation.

You can script backups.

You can script audits.

You can script reports.

You can script remediation.

You cannot script culture.

You cannot automate people into agreeing with each other.

You cannot pip install leadership.

You cannot fix a broken process with a YAML file if nobody agrees that the process is broken.

The technology still matters.

Keep learning it.

Keep building.

Keep experimenting.

Keep showing what is possible.

Just remember that the most sophisticated automation platform in the world still needs something solid underneath it.

A foundation.

A shared direction.

And preferably a crew that has agreed not to eat the oars.