Marketing DevOps and testing within the Enterprise
The state of things
The Aroma espresso was getting colder while my conversation with Guy Weinberg, a colleague and a friend, was just getting hotter. It was a sunny Monday and we were discussing how a few teams in the organization are able to deliver reliable software relatively fast, utilizing DevOps principles and tactics, while others seem to be stuck with slow releases and mostly manual testing and deployment.
By that time Agile and DevOps were standard practices in software development, but since enterprises are slower to adopt new technologies and methodologies our organization was not quite there. The infrastructure was in place but the culture has yet to make the shift entirely.
In our mind, DevOps methodologies did not propagate through the entire organization due to the perceived high entry cost and low ROI.
Various tools and practices seemed to have a steep learning curve, which drove decision makers away. For example CI tools like Jenkins, remote PaaS for testing like Selenium Grid, custom Powershell scripts for deployment or SaltStack configuration for deployment to VMs (in pre-K8S era it seemed reasonable).
As a result, both middle management and senior management, as well as team leaders, were not too eager to start shifting towards building DevOps capabilities. The initial investment just seemed too great and they were fine with the way things were already. It was ironic since delivering a lot of features fast, like they desired, practically requires reliable CI/CD pipelines and a DevOps oriented development and mindset.
Testing was another complementary issue. Teams developing legacy systems that didn’t incorporate testing into their development process years ago weren’t about to start now. Worse than that, developers carried over this bad habit (of not investing in adequate testing) to new projects and the cycle continued.
Teams that were slow would gather once in a few months at night, perform manual UI tests and manually deploy, not without downtime. Since manual testing is a very tedious and painful process for developers — they were not eager to do it often, so business advancement was slow.
The Goal
Our goal was simple — creating a CI pipeline with a single E2E UI test that verifies the correctness of a basic flow of the system in a single work day with our help. Having a safety net in the form of an E2E UI test would be great for the sense of confidence of developers and alleviate the pain of performing manual testing. That, in turn, would increase deployment rate.
We thought that such results would convince developers, team leaders and management to invest time to promote testing and CI/CD throughout the organization.
Tactics
Though simple, our goal was not so easy. The tactic was to create an abstraction layer that would allow the developer to simply write tests without worrying about any of the plumbing or infrastructure involved.
We wrapped the existing Selenium framework for .NET with plug-ins that seamlessly integrate with existing enterprise solutions. Solutions that provided authentication, remote testing PaaS, visualization tools for reports and more. The second article in the series will be a deep dive into the implementation details.
Ready, Set, Go.
Before pitching senior management the idea of spreading this framework organization-wide we needed a test run. After developing an initial version of our pluggable framework we’ve integrated it successfully into our own teams. Then, we needed a single team to do a POC with.
The team leader we approached was excited for the opportunity and the experiment was a small success — in under 24 hours he could run a few E2E UI tests on his machine as well as on a remote platform. The testing project was yet to be integrated in a CI job but it was close enough for a green light for pitching the idea of expanding such POCs throughout the organization.
We presented senior management with the fact that developers do not have adequate DevOps and QA capabilities to support the rapid growth they desire. We suggested focusing on POCs like the one we’ve done, integrating our pluggable framework to cut down costs dramatically. We argued that such POCs were the 80% value with 20% of the effort and that they would at least get the ball rolling.
Unfortunately, we both switched roles before we could see it unfold fully, I can only hope that pushing through with the right tools and tactics would have gotten the job done.
Take Home Message
Firstly, take all of this with a grain of salt. I might be wrong about the reasons that this culture unfolded throughout the organization. I might be wrong about what senior and mid level management thinks about the ROI of testing and DevOps.
However, even if I’m wrong about all of it, I believe that the principles that led me here can help you solve complex and broad problems within your organization:
- Attack the root of the problem — Identifying the perceived high entry cost as the problem allowed us to mitigate that concern.
- Find a pain point to leverage — Developers are willing to put at least some effort to avoid performing manual work
- Provide practical tools for resolving the issue — Using the framework we’ve developed was very practical. The time saving aspect of it was crystal clear.
- Focus on providing 80% of the value with 20% of the effort — Focusing solely on simple CI with a single E2E test, utilizing our framework to save time, resulted in a big boost in confidence, reliability and agility, for a very small price
I hope the lessons I’ve learned about tackling such complex problems will help you in your journey to change your organization for the better.
