The term hyper-automation is widely used in various industries, although some businesses may feel overwhelmed by the processes that come with it. However, the goal of hyper-automation is to simplify, not complicate, the automation of as many business and IT processes as possible. For some, it represents the next wave in productivity, while others fear for their jobs. Yet, while generative AI may seem a bit daunting (think of the recent news surrounding ChatGPT) when used the correct way, it holds a world of promise.
The general attitude across business is that hyper-automation is the answer to advancing business goals while creating the ability to increase output (without increasing staff). But, when you dig a little deeper, the conversation can get more convoluted, especially in the world of Kubernetes.
Most teams have a growing awareness of what is required for a (multi-cloud) production Kubernetes setup. Not just for day 1, but also for day 2 operations. Important factors, such as maintaining compliance, increasing the security posture, and preventing vulnerabilities or, worse, exploits, are extremely complex. On top of this, there are recent studies that have shown that manual work is leading to these vulnerabilities as well as misconfigurations. Considering the world of DevOps revolves around configurations, it’s easy to understand how there can be small discrepancies, thus highlighting the need for automation.
Often, we see that business needs are not aligned with the needs of the individuals, which in this case, refers to the Engineers. From software engineers to infrastructure engineers, they all look to solve complex problems, as this is where their talents lie. However, the difference here is that infrastructure does not differentiate a business. Software does. Due to this, companies will always look to find ways of simplifying and automating infrastructure. While engineers are also fundamentally looking to do the same, they see themselves as responsible for building these systems to automate. So what if there are solutions on the market to help simplify and automate?
Most companies will look at the big picture and directly compare doing the work themselves versus automating the work. Naturally, this may come as a threat to engineers performing the work. As a company, having these opposing interests does sometimes come at the expense of the engineer, and this is oftentimes why these engineers can be combative towards these changes. We often hear arguments such as; “We can’t use that technology in our specific setup” or “our current architecture and/or way of working would not align with the technology”. The difficulty for managers is that engineers often have in-depth knowledge about the environment and requirements, whereas these leaders don’t have that level of domain knowledge. Therefore, it's a challenge for these leaders to have the required understanding to push back toward automation and standardization.
Anecdotally: I had a conversation with a freelance engineer a few weeks back working for a large bank setting up a system atop of a public cloud. They were now almost ready to get the first app in production. He operates in a team of 5-7 people and was really proud of what they built. The work seemed tremendous, and I definitely believe he should be proud of the achievement because it appeared to be a very complex system. However, the goal of that project was not to build cloud infrastructure. The goal was to get an app, or apps rather, in production. The downside was that this project had now been going on for 3 years due to its complexity and manual work. When we talked about automation and standardization with technologies available, the answer was exactly that: “The solution would not be a fit because our application has specific requirements”. If we do the math on this project now, even before the first app is live, I would doubt whether the same decision would have been made at the start. I know the cloud world is evolving at tremendous speed, so I know that some of the current technologies were not available 3 years ago when this specific project started. Also, it’s hard for people to change course once they have come such a long way in what they set out to achieve.
What fascinates me is the fact that companies who need to start the journey today are in the exact same situation, and some might take the exact same decision to build technology stacks when there are viable products out there to evaluate. Managers and leaders are then faced with the dilemma: how do I satisfy the needs of my very capable engineers that I want to keep happy and motivated, yet automate and standardize as much as possible? To drive value for the company, the question should always be; what to engineer and what not to engineer. There is not a single answer to this question. Still, in general, I believe that most of a business’ investments and capacity should be geared towards what differentiates the business from its competition or, in the case of government, does it drive value for the user. Building a complex cloud system with an even more impressive Kubernetes architecture can be an achievement, but how does that provide value for a business or governmental institution? If viable solutions exist on the market to solve a problem that does not differentiate you, consider adopting those solutions.
In the Public Cloud and Kubernetes Ecosystem, companies use these technologies to speed up the software development lifecycle and deliver products faster. However, some projects can take a long time due to their complexity, and engineers may choose to build their technology stacks instead of adopting available solutions. Companies starting their journey today may face the same situation, and managers must find ways to automate and standardize while keeping their engineers happy and motivated. The focus should be on differentiating the business from its competition, driving value for the user, and centralizing security, compliance, automation, and operational activities.
Platform Engineering can help offload the operational burden from developers, but the goal should always be to engineer what adds value to the company. Companies should document their end goals for the platform, understand the needs of their developers, and prioritize according to business needs. They must integrate and configure technologies to accurately match their needs as different businesses have different requirements from software engineers and use different software stacks.
In my view, the evolution of DevOps to include Platform Engineering is a very welcome one because it implies there is an acknowledgment with businesses to centralize some of the security, compliance, automation, and operational activities that companies need to control and enforce. Secondly, it offloads some of the operational burden and cognitive load from developers. For many development teams, the activities are similar, so synergies will be gained from centralizing, allowing for development teams to spend more time on coding and driving business value.
But, I have still not addressed how to address this dilemma, as we see the same thing happening in these Platform Engineering teams. The goal is to deliver a secure, compliant, automated developer platform in order for developers to be more productive. In achieving that goal, the question should still be what to engineer and what not to engineer. If a company intends to automate pieces of that platform, that could be less appealing and intellectually challenging for the engineers.
The first thing could be to make sure there is a clear connection between the goal of the platform team and the goal of the company. Too often, we see engineers getting an assignment of “we need to have a platform for developers, can you build one?” The pitfall here is that platforms often get engineered inside out. So engineers start with the foundation as opposed to the actual users. If there is a clear connection between company goals, developer goals, and thus platform goals, it becomes more apparent to engineers what is valuable for the company and thus allows them to prioritize according to the business needs.This makes the choice more clear for engineers what to take on by themselves and what readily available solutions to implement.
The second thing is to clearly document the end goal for the platform, with regards to capabilities, to get to the best developer experience. Note: in-depth conversations with developers (users of the platform) are required to understand what their needs are. By thoroughly understanding these needs you can then truly understand the components required for success (whether this be technology or manpower). While technologies used for mass adoption are standardized, which in fact is useful, they still require integration and configuration to accurately match the needs of a company. Development, security, and compliance processes differ from business to business, and the technology used must reflect this. Requirements from software engineers will differ and the software stack will differ. This implies that the tech you ‘buy’ will always be a semi-conduct and will have to be integrated into the processes of the company. Elements will have to be added to get to the required self-service capabilities for developers. Aligning company goals and talking through the value add engineers have in achieving those goals with the end product, not the semi-conduct, can relieve the intimidation and competitive pressure from externally sourced solutions.
