Written by Ben – Senior Network Engineer

There is a saying – “Change is good”. In the context of technology, change represents risk. Risk appetites vary, but when there are an organisation’s finances and reputation on the line, the risk appetite is generally low and conservative. That being said, change must happen in order to improve, move forward and resolve problems. The balancing act of risk versus reward, pros versus cons, and benefits versus drawbacks is not a new concept. They are questions in the forefront of the minds of managers and senior leadership staff, right up to CEO level. Engineers often have to assess them too. “Change”, “risk” and “benefit” are three of the most commonly used terms across today’s ever-changing IT industry.

Here at Opticore, we not only witness but enable change on a daily basis, both internally and for our clients. We are an evolving business; redefining and reshaping the way we do things to meet new customer requirements, helping them to evolve in the process too. We’ve led and supported change of all shapes and sizes in many organisations across a variety of industries. We’ve played a key role in making networks more efficient, easier to support, cheaper to run, resilient, up to date, the list goes on. Change doesn’t scare us like it does others; it enthrals us and we grab opportunities with both hands to help make it happen. As a result, the opportunity to learn lessons based on our experience and observations are vast.

This article does just that, and focuses on a subject that is common to our many clients across multiple sectors – change implementation. It’s a hot topic, everyone does it differently using a variety of approaches, tools and systems. At a deeper level, the text discusses a manual change implementation approach that is primarily driven by human action and traditional CLI working methods, versus an automated approach assisted by Infrastructure as Code (IaC). It draws upon first-hand experience in designing, deploying and using, an automated change implementation system, as well as transitioning onto it. It does this by presenting the key advantages of such an approach, which are already well known and widely publicised, as well as the challenges organisations can expect to face in doing it.

The latter is where we feel readers can benefit most. All too often decisions in technology are made without considering what we refer to here as the “bit around edge”. In other words, the complex, non-technology related challenges that need to be overcome in order to make such a change happen. There are many, however we’ve picked the ones that we think are the most important to consider when adopting automation and IaC as the preferred change implementation workflow.

Benefits of an Automation & IaC Change Workflow

The introduction of IaC and pushing changes using an automated workflow brings with it many positives. The theoretical benefits are already well documented elsewhere; however, we’ve provided the main ones we’ve witnessed first-hand at one of our clients in the financial sector here.

This section, as well as the one succeeding it, draws inspiration from this client. We’ve tried to mask specifics as best as possible out of not only confidentiality, but respect as well.

Reduced Human Error & Ambiguity

Human error has long been a problem in technology. Humans are intelligent beings but even the best can still make mistakes or overlook things. In the context of a manual change implementation workflow, or any manual workflow executed by humans for that matter, there is a reliance on variables attributed to the human psyche. Interpretation, understanding, awareness and attention are just some aspects that can have an influence.

When it comes to change failures, a large percentage can be attributed to human error, specifically due to the variables just mentioned. With an automated workflow, some of these are removed, either partially or entirely, meaning the risk of change implementation falling victim to human error decreases.

Mistakes and typos can now be spotted a lot earlier during change preparation by running pre change tests in a dedicated test environment which mimics that of production. Bugs can also be identified well before change execution as opposed to during, avoiding potential sticky situations and escalations at 2 o’clock in the morning. When it comes to running pre checks just before change execution, the tools can provide an element of cover here too. Determining that a particular pre change condition is an issue is no longer solely on the implementing engineer. He (or she) also has the tools backing him up to identify problems that he may have missed, a safety net if you like.

Copy/paste errors or fat finger moments during change execution and backout can have a big impact. With a workflow that incorporates IaC, the risk of such instances is significantly reduced. Everything is templated and validated in the code prior to being pushed into production, meaning risk is reduced and an increased comfort level around change implementation in general can be felt.

Traditional post change checks relied heavily on a human to sift through configuration differences highlighted by a comparison tool such as Beyond Compare. The engineer would then either acknowledge discrepancies as conditions that were expected or take remedial action if necessary. This left room for subjective interpretation of what was and was not deemed as an issue and placed trust in the engineer working through any verification differences to take the correct action. Shifting some of the onus off the engineer and onto the tools reduces the possibility of something important that is difficult to spot being overlooked, or an engineer not taking corrective action in response to a condition that doesn’t look right post change.

Diminished Configuration Drift

This is one of the hardest things to control in organisations with hundreds, if not thousands of devices that are managed individually. Even with the best build and standards documents, configuration drift in such an environment is inevitable. As a result, it can increase the resolution and recovery times of incidents, delay BAU activities, and make projects harder to navigate and more expensive to deliver.

With a network that is managed as code and modified using automated workflows as opposed to manual, configuration drift effectively becomes a thing of the past. Because the network and how it behaves are now defined in template files that are applied to groups of devices dependent on their use case, this essentially eradicates any configuration discrepancies. For example, the way in which a VLAN or SVI is created and the parameters they use is now pre-defined, meaning there are no configuration inconsistencies across devices or even on a single device, such as a difference in the way a particular SVI is configured compared to another.

The near abolition of configuration drift contributes to stricter adherence to build and configuration standards, easier operational supportability and BAU job completion, faster incident resolution, and better overall configuration consistency across the network estate.

Consistent Implementation Approach

In the manual days, everyone had their own way of doing things. Usage of different terminal programs, various ways of conducting pre and post checks, dissimilar methods of capturing implementation logs, and implementation and backout plans that used varying formatting styles containing content which differed greatly from one plan to the next. These are just some of the facets of manual change implementation that lacked consistency across the engineer pool.

The above often made it difficult to quickly sift through logs in the event of an incident suspected to have been caused by a change, or peer review an implementation and backout plan. This is because there was no baseline expectation. People didn’t know what to expect when they went to open a file and would need time to familiarise themselves with what was in front of them. In many sectors and indeed Finance, time is money. 10 minutes could be the difference between no loss, a £100,000 or even a £1,000,000 loss. Reputation is also paramount; an unwanted appearance in several national newspapers the next day could be catastrophic. Half-hearted attempts to introduce templated documents and SoPs often fall victim to drift and lack prolonged enforcement as old habits creep back in.

Pushing changes as IaC does away with all that. Being able to quickly review an implementation and backout plan whilst at the same time being able to be thorough brings with it a certain amount of fluency and slickness. Engineers end up implementing in pretty much the exact same way as one another using the exact same method, guided by the various automation toolsets in which to do so, rather than using their own distinct technique.

Across all team sizes, especially large ones, an automated change implementation workflow helps tighten the leash and remove an element of doubt associated with change implementation. Knowing that everyone is following the same method and working to the same principles again brings comfort and helps reduce change risk.

Reduced Implementation Time

Humans can only really do one thing at a time. Being able to implement on multiple devices at once simply isn’t possible using a manual workflow without more than one engineer being involved. This means that a manual change implementation approach is linear in nature – step 1 must be actioned before step 2, step 2 before step 3, and so on. Now, tools leveraged by automated workflows can execute configuration pushes on multiple devices at once, saving significant amounts of time.

Large-scale implementations that previously took hours can be completed in as little as a matter of minutes. Long gone are the days of initiating an SSH session to twenty devices via PuTTY and having them time out on you, only to have to enter your credentials every time you want to verify or implement the next step on a device, potentially causing you to lose your logs if you didn’t set the logging parameters correctly. Implementing using automation and IaC means most of the work is now done during business hours, resulting in the amount of work required during change implementation being substantially reduced. Engineers prepare the code, test it and weed out issues, test it again until it’s good to go and get it in a position to push (but don’t actually push it) inside business hours, then turn up out of hours and quite literally “click the button”. This is not to say that in doing so engineers simply walk away after clicking said button and that’s it, job done. There is obviously more to it than that, but change implementation shifts to becoming what is, in many cases, a babysitting exercise, as opposed to being a fully-fledged, all-encompassing activity that requires 100% focus and involvement for the duration.

One of the major benefits of reduced implementation times is the positive impact it has on employee happiness and wellbeing. Implementations that take less time mean engineers ultimately end up working less, which benefits the business as well. A happier workforce is a more productive workforce.

Keep Pace with the Industry

The word “automation” is no longer a buzz word. It’s happening all around us: at work, home, when we travel, even when we sleep. Being able to keep pace with not only the uptake but also the usage of automation is something that organisations must do. It is simply a non-negotiable in 2026. There may be bit-part automation already happening in pockets within departments, but migrating to an automated change implementation workflow can really turns things on its head and do away with outdated and cumbersome manual approaches.

Being able to attract top talent from external is a factor too. Engineers want to be working on the latest and greatest, not typing “conf t” on a 6500!

Considerations

There are without a doubt, many advantages of transitioning to an automated change implementation workflow that incorporates the use of IaC. These are well documented both in theory and real-life examples as we’ve just seen. But what is less widely available is information around the challenges that IT departments face when trying to make it happen. These challenges differ across organisations and can vary significantly across sectors as well. Every business has their own way of doing things and particular rules that they must follow, which bring with them organisation specific as well as common complications that must be navigated.

We look at the challenges our client faced when adopting automation and IaC as the preferred change implementation workflow below.

Mindset/Culture Shift

Old habits die hard. In a large team of engineers, some of which may have been around for decades, they die even harder. People can be resistant to change out of fear of the unknown. Amongst other things, it takes time, effort, money and perhaps most of all, patience, to change the way people do something that they’ve done for years. “Normal” methods become second nature and thus people become conditioned.

Mindset, both individual and collective, is not tangible, so is a difficult thing to change at all, let alone quickly. Remember also that people, unlike what we’re discussing here, are not machines. They need to be afforded the time, space and opportunity to learn, at different rates whilst being able to use a variety of methods. Redefining the status quo by breaking the cycle and instigating a mindset shift across a large team of engineers is challenging and requires consistency, tenacity and commitment in order to achieve it. Arguably, an element of faith is needed as well.

The mindset shift is not only needed at the engineering level; it’s required at management and senior leadership levels too. Support from above is key, and making sure everyone is on the same page is pivotal. Depending on where the initiative or decision to transition has come from, there may be a bit of convincing to do. Internal politics can play a big part in medium- and long-term strategic direction and even day-to-day ways of working. There likely isn’t an endless pot of money for everyone to help themselves to either, so some may need to hone their powers of persuasion to get things moving.

Ultimately it needs to come from the upper levels i.e. management and senior leadership. It can’t just be a flash in the pan, some idea that’s been thought up and they are winging it. There needs to be structure, purpose, things put in place that show and demonstrate to people that things are about to change. There needs to be constant and consistent dialogue, keeping people in the loop, updating them on progress of development etc. It can’t just be something that’s mentioned once or twice then nothing at all as people will lose interest and you’ll have to warm them up all over again. People also need to know what training they can expect and be reassured that they won’t be thrown in at the deep end.

Staff Training

A shift in mindset/culture, at the engineering level at least, cannot properly start without adequate training. People need to see it to believe it, either vicariously through others or by doing it themselves to really initiate that shift. Engineers will have different learning styles, learn at varying rates and have individual strengths and weaknesses. Some can do or be told something once and recall it, whereas others need four of five attempts plus guidance before they’re fully comfortable doing it by themselves. Some will just “get it” and some will “sort of get it”, the latter being enough for them to get by and do what’s necessary at a base level to achieve a desired outcome. Anything beyond that and they will often fall back on one of the “get it” people and use them as a comfort blanket. And that’s OK; not everyone can be a superstar or a “know it all”. There are always going to be people that lead the charge with change and people who follow. This is true in all areas of life, not just technology.

Delivery of training needs to be structured, organised, varied and accessible, with those delivering the training and the training resources themselves accessible as well. Webinars that are recorded and made available for consumption, drop-in sessions (clinics), documentation repositories (e.g. SharePoint/Confluence), live demos and group chats are all examples of training delivery and resources that should be considered. Understanding that the engineer pool is made up of people with different personalities and characteristics, in particular learning styles, is key to ensuring the uptake of the new way of doing things is as efficient as possible.

Engineers aren’t the only ones that will require training; management and in some cases senior leadership staff will need upskilling to a certain degree as well. These are the people that review and approve change implementation to proceed, so they will need to be able to at least understand at a base level what the new approach entails and what the end-to-end process flow is, so that they can make informed decisions.

People Single Points of Failure

An automated and IaC-centric change implementation workflow brings with it a lot of new systems that are interconnected. Bear in mind that there are relationships between these systems; numerous dependencies that must be operational for the end-to-end implementation process to complete successfully. The more tools and dependencies there are, the more chance there is of something going wrong and not working as expected. The individual systems all have their part to play but cannot work in isolation.

There will probably be only a small number of engineers who truly know how the automated IaC process works and what is really happening under the hood. These will probably be the engineers who set things up initially and have/are actively delivering training to the rest of the engineers. These people can end up becoming single points of failure if a particular component in the complex automated system of tools and dependencies breaks – “if John is off then we just have to wait until he’s back”. Think about it, 100+ engineers, all new to automation and IaC with little to no experience and only a small amount of training under their belt, pushing changes, sometimes multiple on the same evening. This has the potential to create a bottleneck and an ever-increasing reliance on the people aligned to support and escalate to in the event of issues. Cue employee wellbeing… or rather lack of. This is a real (and realistic) problem – all eggs well and truly in one basket.

Network Engineers spend a large amount of their time avoiding, identifying and remediating single points of failure. Wouldn’t it be ironic if a Network Engineer themselves ended up becoming the thing they work so hard to prevent? It’s imperative that people single of points of failure do not exist or get introduced. Reliance on one person or a very small pool of people can lead to the new workflow falling flat on its face, disrupting operations, delaying project delivery and causing substantial business impact. Questions will no doubt be asked at senior leadership levels too, which could end up meaning support for the new solution starts to wane.

Strict Regulations, Audit & Compliance

The finance sector is subject to strict regulatory requirements that mandate certain conditions must be met and that there is a valid audit trail for pretty much everything. The nature of the financial institution and what it specialises in will dictate the exact regulations it needs to comply with, which can potentially make introducing new systems such as this difficult to navigate. The hoops may be so numerous that it may not be worthwhile or even possible. An understanding of the exact internal and external audit and compliance requirements is fundamental, as is knowing whether they can be met with the new system, both initially and continually. This information is critical to deciding whether the solution is viable or not. Discussions around this should take place from the outset.

Regulatory requirements would include things like encryption, disaster recovery, resilience and maximum down time SLAs. Audit requirements would be things like change management and elevated access. Compliance requirements can vary and tend to cross over with regulatory but could include other things like PCI DSS, mobile and online banking standards.

Existing Complex Technical Landscape

Many large organisations have large networks. These networks can often be complex and operationally fragile, expanded over time through acquisitions and mergers, with an ever-increasing number of skeletons in the closet just waiting to wreak havoc. It can often feel like a game of Jenga; move (change) one piece and risk the entire thing falling over. Understanding the existing network estate is vital to envisioning how an automated workflow could be positioned and where any potential new supporting infrastructure can be placed. Dependencies on existing systems should be highlighted, as should any integration that may be required.

Once commissioned, migrating from old to new and flipping the script from manual to automated brings with it challenges that will likely require thinking outside the box by the brightest minds in the department. There will be physical as well as logical challenges as well. IaC may be facilitated in software, but that software needs to be hosted from somewhere. Then there’s the question of resilience and redundancy – what happens if component A fails in location B? What’s the fallback position and where does it leave us if the whole thing breaks? Can we fall back to using a manual approach if necessary? These are all valid questions that warrant serious thought.

Vendor Support

If new infrastructure is to be deployed and configured using IaC, support from the hardware vendor is highly beneficial and recommended, although not absolutely necessary. It will certainly help speed up the deployment as the vendor will likely have experience supporting other organisations to achieve similar outcomes. This experience will offer valuable lessons learned as to what and what not to do at key junctures and decision points.

Having the vendor engaged can make deployment and operational issues easier to resolve along the way and keep achievement of project milestones on track. However, there will likely be a charge for vendor support, the costs for which will need to be factored into the viability of the transition. On the other hand, if the vendor is not engaged, the organisation opens itself up to difficulties with deployment and migration, as well as day-to-day usability issues. With nobody to escalate to beyond internal senior Network Engineers who may not have the skills to remediate, this can quickly become a problem.

Consider as well that if the hardware being supplied is from a vendor that is completely new to the business, a relationship will need to be built from the ground up. This relationship will need to be one that is founded on trust, which can take time.

Time

Time is money. Moving to an automated workflow will likely take lots of both. Being able to dedicate people to developing the necessary architecture, toolsets and documentation, as well as deliver training and providing ongoing support represents, at least in the short-term, a big upfront fee, the benefits for which may only be realised two, maybe even three years down the line. This is why it is vital to make sure that everyone is on the same page and is committed to the cause, particularly those who are key decision makers at senior leadership levels. Migrating to an automated workflow for change implementation is, for most, delayed gratification personified.

Ensuring the people initially aligned to the transition continue to be is important too. All too often, engineers are pulled off projects and assigned to other pieces of work that are deemed higher priority, or get bogged down in rudimentary operational issues that drag on for long periods of time. Safeguarding a large percentage of engineer time so that it is spent pushing the transition forward is paramount to enabling the timely and efficient delivery of the system. There will of course be instances where time must be spent on the more complex operational issues that have been escalated; the transition is, afterall, a project, and operational stability must come first. But being able to hand back the reins once the problem is fixed and refocus on the primary task at hand as quickly as possible is key. Assistance from the management level will be required to do this effectively, and there will be a certain degree of ringfencing around specific engineers that will need to be built to protect their time.

It’s not just the delivery of the automated workflow that will take time. The upskilling of the engineer pool will be time consuming too. As mentioned in the Mindset/Culture Shift section, patience is an important aspect as well. There will no doubt be times, most certainly when engineers first start to adopt the new approach, where things don’t go to plan or aren’t implemented as quickly as they once were even when following the manual method. This can frustrate senior leaders to the point where questions are asked, which is why management need to remain on side and provide a degree of cover for those who developed the system and use it every day. Difficult conversations may need to be had, and certain people may need to be reminded of the benefits that are on offer further down the line.

Cost

Underpinning all the above considerations is perhaps the most talked about subject in finance – cost. “How much will it cost?” and “How much money will we save?” are probably two of the most common questions asked when discussing changes to the status quo. Internal politics will play a part, and anyone selling the idea of an automated change implementation workflow will not only need to sell the idea but everything else that comes with it.

It’s possible that the existing hardware may need to be upgraded or even replaced, which carries both significant initial and ongoing investment. Migrating onto that new hardware is costly as well and may require involvement from other teams across the organisation to help facilitate it. The migration will need to be managed as a project and have people with the necessary project management skillset to do so.

Ensuring the right calibre of engineer(s) driving the development and transition are present within the business is critical to getting things right first time and reducing the headache that it will inevitably bring. If there aren’t suitable people already present within the business, going out to market and recruiting external may be required. This can be costly and will delay delivery as newbies will no doubt need to navigate a potentially cumbersome onboarding process.

Conclusion

In technology, it’s often the technology itself that generates the most debate. The subjects of cost and time often receive the most air and screen time outside of this, but other factors are significant contributors that require careful contemplation that often get forgotten about.

This article is not intended to put people off migrating from a manual change implementation workflow to an automated one that leverages IaC. Nor does it in any way intend to criticise automation, IaC, anything or anyone related to the real-life scenario it draws insight from. What it aims to do is call upon first-hand experience and present the challenges associated with such a transition as factors that should be considered. The observations laid out are ultimately there to provide guidance to key decision makers within organisations that may be considering the adoption of an automated workflow to replace an existing manual one. If a decision has already been made to hit the button and proceed, the information is pertinent to these organisations too. Some elements may have already been discussed at length as part of “Go” decisions, but the hope is that it helps identify gaps and make businesses ask themselves, “have we thought of the bigger picture?”.

Moving from a manual to automated change implementation workflow first needs to be viable to even be on the page for consideration. Does the organisation implement enough change to warrant the initial and ongoing investment? Numbers will need to be crunched, and reliable data will need to be available that identifies trends in areas such as volume of change, total and average time spent on implementation and backout, number of incidents attributed to change implementation, and the total and average time spent working on fixing them. Following this, an understanding of how these statistics compare to accurately projected figures associated with an automated workflow are required. Once these numbers are known and the potential benefits are identified, the bits around the edge then need to be considered.

Cue the challenges outlined in this article and the impact they could have on the viability of the solution. What (if any) are the problems the organisation is experiencing with a manual change implementation workflow? What hurdles will they need to jump through to not only get it up and running but maintain it? Do they have the right people to be able to do it? Are they able to commit engineers to developing the system and training the rest of the engineer pool? Is it worth it? These are just some of the many questions that will need to be asked during proposal meetings and round table discussions. Once all of this is considered, only then can a suitable, informed decision be made.