Have you ever wished you could change your system and immediately know if said change had the desired effect? Proper testing is paramount to system administration. Whether it’s new functionality, code changes, or configuration changes—this usually necessitates full promotional-path testing. Typical testing methods are mostly manual, which is time and resource intensive. Well, I have good news for you, because implementing automated testing for your system may very well be the answer you seek!
The Proof is in the Pudding
Back in 2014, we had a sizable client (around 2 million customers) going through a significant ArcFM/Responder upgrade. Our “ecosystem” was comprised of around 50 machines with several integration systems interacting. In a previous upgrade, system changes took days to test, and even longer to have resources provide the data necessary to ascertain if performance targets were being hit and what may need further changing/testing. Bug testing fared little better. Clearly a new plan was needed for the next upgrade and Schneider Electric (SE) did not disappoint.
We began by harnessing the power of a continuous integration (CI) platform as a central controller, and utilized software that allowed for repeated automated actions to take place. Leveraging the control mechanisms inherently built into CI platforms, we were able to create one-click tests that did a variety of steps—actions that would typically be extremely time and/or resource intensive.
- Stand Up Steps—putting the system in a “ready” state
- This may seem trivial, but this includes restoring/restarting databases and VMs (over 50 machines), cleaning up hundreds of virtual test users, checking out and applying the latest code from subversion, and creating new virtual users that sit and wait to take actions.
- Run Steps—running all necessary tests
- Here we would start performance counters, begin a script that triggered hundreds of virtual users to execute on-screen actions, and start our test tools (these consisted of tools that would record, and simulate a storm, outages, and customer calls).
- Tear Down Steps—stopping everything and collecting data
- After a predetermined amount of time for a test (some as long as 24 hours), we needed to end our “action” scripts, collect all data accumulated throughout the test (this mostly consisted of performance and functionality metrics), and cleanup any outstanding virtual users and queues.
We were able to develop a system where a single resource could execute a test that, in some 20 steps, would accomplish the actions and responsibilities of hundreds. Moreover the framework immediately provided test metrics--anything from system and database performance to functionality metrics.
A Wider Vision
The previous example, while focused more on performance testing than anything else, gave us a glimpse of what could truly be accomplished with the right mindset and technology. Our clients have a magnifying glass on performance these days, and SE wants to be the resource utilized to help highlight issues and provide solutions. The beginning of that is simply uncovering pain points in a more efficient manner—this means automation.
Many solutions exist regarding automation. We feel that the proper approach is to maintain a control platform that can be totally customized to suit any testing needs. Frankly, we have little option in the matter. Our clients’ needs are extremely broad and often times are vastly different than one another based on size, geographic location, and government/business rules that dictate processes. Remaining flexible in our approach allows us to provide utilities with a customized solution that fits perfectly into their system, taking into account all processes and integrations.
In a perfect world, we would gather all utilities, great and small, and, utilizing input from like-concerns, create a set of test scripts that can be universally applied and run by any client. In this vein, we could implement like-automation across many clients. With this approach, analysis of gathered metrics could universally benefit, not only client-side performance, but be brought back to our Product department for further R&D improvements. While client Application Metrics (a separate aspect) are provided in 10.2.1b for our Product department, they are not nearly as all-encompassing as system tests.
Okay, but why?
This all sounds well and good, but some of you may not face the scaled challenges of a large utility. You might be reading this and thinking, “We don’t have testing or performance problems.” While it is true that some of the smaller utilities do not face a multitude of performance or functional testing hurdles as your large utilities do, one must not discount the value of quick turnaround in any exchange.
Take a code change, for example. Let’s say Small Utility Power (SUP) has a new functionality they need added to their customizations. SUP and SE strike up a contract to have SE develop this new functionality which is then delivered to SUP. Typically, regardless of size, a utility is not quick to rush some new functionality into production. Likely, they will install in a lower environment, like a sandbox, dev, or QA, and then run a series of tests to determine if the desired functionality not only works, but adheres to intended design.
As this new functionality is promoted into production, there is typical back-and-forth regarding bug fixes or additional tweaks to the software. This stage can be extremely time-consuming and can often be very difficult to identify what the problems are, or more frighteningly, what problems may come. It is precisely at this stage where automation is critical. With the introduction of automation, SUP could receive new installers, drop them in a folder and begin a test with a single click. Immediately, feedback could be provided just by looking at pass/fail criteria. Moreover, this same pass/fail test-script could be used during development to make the project itself more efficient. Could you imagine how much time you might save by running automatic functionality tests?
Our Internal Use
We don’t just believe in automation implementation because it is good for the utility industry, we implement automation in as many facets of our internal infrastructure as possible. For example, during our debug phases of a project or bug fix, we will often make code changes and have the need for rapid testing. We can test on the fly within our development environment, but that doesn’t always capture the client’s experience. It is imperative that we deploy our new installers to a “pure” test machine (a VM that mirrors a client’s environment).
Normally, it would take a decent chunk of time to commit code changes, build installers, uninstall old installers from the VM, install new installers on the VM, open up programs, run the test, and analyze the results. In fact, sometimes a single instance of this cycle could be a day or more. With automation, we are able to commit a code change, and all the rest is done automatically—with the choice of running test scripts for further functionality testing.
A Call to Arms
Clearly, we can all get along without automation. Sure, it’s nice, but is it imperative to your business? For many, after demonstrating what automation can do for their ecosystem, the answer is yes. If you could consolidate days to hours and hours to minutes, you’d be amazed at how many more cycles can be iterated through without further increasing resource cost. Once this efficiency is injected into your business, it becomes impossible to live without.
Remember that “perfect world” I wrote of earlier in the post? The cool thing is this can actually be achieved! We are already working with some folks who are big on driving such changes, and together we truly do feel we can revolutionize how we all work together. Though it is true not all utilities experience the same problems, all utilities unite over commonalities in workflow. I urge you all to invest R&D into automation. Be pioneers!
Schneider Electric is dedicated to building, deploying, and assisting in maintaining automated frameworks to suit very specific needs. Remember, we are your partner and are driven by your success. Whether it’s consulting or developing, SE can walk with you and your business into the future. Together, we can improve this industry and revolutionize your processes!
Please contact your Sales Account Manager for further information.