Continuous Integration for webMethods – How to avoid the speed bumps and when to get off the CI road.

Continuous Integration can be a powerful tool in custom software development. It can also be very effective on enterprise applications like SoftwareAG’s webMethods. However, Continuous Integration should be a tool in your toolbox… but it is not a one-size fits all solution.

Before everyone starts piling on saying how crazy we are, let’s talk definitions.

Continuous Integration does not equal Continuous Testing… nor does it equal Agile.

“Continuous Testing” is a software development approach whereby users’ changes are tested as soon as possible to provide immediate feedback. Ideally, this would also be an automated process. We HIGHLY recommend the use of continuous testing for both the near term and longer term benefits when it comes to regression testing and application maintenance. Simply put, if you’re not actively testing on your project, then you’re throwing money out the door. (See, we’re not so crazy, so you can keep reading.)

“Agile” is a software development approach characterized by short planning and development cycles. Larger overall requirements are broken down into smaller deliverables which can be created during short development cycles (i.e. “sprints”). The idea being that you’re better off designing and developing in short increments. We’ve seen some customers get value from using an Agile approach. We’ve also seen some customers (especially those with larger and more complex requirements) struggle to use this approach to make big gains. We’re not looking to get into a pro/con Agile debate here… just trying to put some fenceposts around a definition as compared to Continuous Integration.

You can use Continuous Integration with Continuous Testing or Agile methodologies but it is not the same thing as either.

Continuous Integration (“CI”) is something different.

By definition, CI requires that code is “continuously” merged into a baseline at the time of commit. In CI, there is no such thing as multiple branches to isolate newly developed code. There are no additional branches so there is also no need to merge branches together at a later time.

Not having to manage the merging of branches down the line is VERY appealing to many people. However, there is a reason branching was designed in the first place… because there was a “need” for it. Developers were having problems stepping on each others code changes so someone (not getting into a historical debate here) came up with branching as a strategy. So, if it is “needed”, how does CI get around branching?Successful Example of CI on webMethods

One approach to getting around the need for branches is to be very careful which webMethods projects will make use of CI. A great example of a project that would be an ideal target for CI would be an API development project. If you’re using SoftwareAG webMethods for this, then a large portion of your work is done in webMethods Integration Server. In this assumed project, your API are all essentially standalone functions. This means that one developer can go into their API code without worry about their changes impacting the other API developers. If these API are all part of the same project, then it is easy to check into the same line of code because, architecturally, the code is already isolated to some extent. You could even consider them their own individual projects. They just happen to co-exist in the same trunk.[one_sixth last=”no” spacing=”yes” center_content=”no” hide_on_mobile=”no” background_color=”” background_image=”” background_repeat=”no-repeat” background_position=”left top” hover_type=”none” link=”” border_position=”all” border_size=”0px” border_color=”” border_style=”” padding=”” margin_top=”” margin_bottom=”” animation_type=”” animation_direction=”” animation_speed=”0.1″ animation_offset=”” class=”” id=””][/one_sixth]

Figure 1: An API Project with isolated web services can benefit from Continuous Integration approach.

Enterprise applications like webMethods are great because they perform specific core functions out-of-box, make it easy to customize the out-of-box functionality, and typically provide integration points that allow them to share information with other enterprise applications. Some of these benefits provided by enterprise applications are also part what makes them unsuited for using CI as a development approach. As an example, we have seen users often run into problems using CI on webMethods projects like the one outlined in Figure 2. (Note: We’re using webMethods as an example in this blog but we’ve seen the same challenges pop up using CI with other enterprise applications.)[one_sixth last=”no” spacing=”yes” center_content=”no” hide_on_mobile=”no” background_color=”” background_image=”” background_repeat=”no-repeat” background_position=”left top” hover_type=”none” link=”” border_position=”all” border_size=”0px” border_color=”” border_style=”” padding=”” margin_top=”” margin_bottom=”” animation_type=”” animation_direction=”” animation_speed=”0.1″ animation_offset=”” class=”” id=””][/one_sixth]

Figure 2: Enterprise applications can run into challenges with Continuous Integration due to the many interdependencies. The application above accepts incoming Purchase Orders as XML (using webMethods Trading Networks)… routes them (using a combination of webMethods TN Profiles and webMethods TN Processing Rules)… submits it for processing (first to a webMethods Integration Server service for transformation and mappings to a standard internal format)… and that in turn submits it to a messaging queue (over webMethods Universal Messaging) which gets routed to a business process (using webMethods BPM)… and, based on some criteria in the BPM process, eventually gets queued up in a portlet for someone to approve (using CAF on MyWebMethods). Underneath the various steps in this process, it leverages several of the API that were developed in the original application developed using CI. Change in one part of the overall application can heavily impact other parts of the application.

The first relates to the interdependency between changes made by multiple developers working on the same trunk. Even if all of these different components were written in Java, you will run into the issue that there is a significant amount of dependency between the various components. Incremental changes made by each developer often breaks other parts of the application which causes ongoing, but inconsequential, “noise” during development. Basically, with enough churn, everything is always broken so you’re not really getting any meaningful information back from tests. This is not a new criticism of CI.

This fundamental challenge is augmented on your webMethods projects by the additional challenge that a single change (whether it is a feature or a fix) often requires tying together multiple technologies to create a single logical feature. However, one single developer will not be working across all of the different products. This means you really need a way to coordinate the various technical specialists activities across those multiple technologies.[one_sixth last=”no” spacing=”yes” center_content=”no” hide_on_mobile=”no” background_color=”” background_image=”” background_repeat=”no-repeat” background_position=”left top” hover_type=”none” link=”” border_position=”all” border_size=”0px” border_color=”” border_style=”” padding=”” margin_top=”” margin_bottom=”” animation_type=”” animation_direction=”” animation_speed=”0.1″ animation_offset=”” class=”” id=””][/one_sixth]

Figure 3: A single ticket (for either a feature or a fix) can get assigned to multiple developers who now have to coordinate their activities. In CI there is no easy way for the developers organize their code in a separate branch. In addition, as they develop and make changes, they continuously break each other’s code which creates a fair amount of “noise” in the CI process. (Note – This is probably why they all look so unhappy in the image above.)

[one_sixth last=”yes” spacing=”yes” center_content=”no” hide_on_mobile=”no” background_color=”” background_image=”” background_repeat=”no-repeat” background_position=”left top” hover_type=”none” link=”” border_position=”all” border_size=”0px” border_color=”” border_style=”” padding=”” margin_top=”” margin_bottom=”” animation_type=”” animation_direction=”” animation_speed=”0.1″ animation_offset=”” class=”” id=””][/one_sixth]Most importantly, if you’re committing all changes into a single baseline, then you have no easy way to break out one developer’s changes for a feature from another developer’s changes for a fix. What do you do when a production hot fix needs to get into production immediately but another developer’s changes for a feature are breaking the current build?

Feature Toggling (i.e. “Branching for CI”)

Now, CI proponents generally try to address this particular challenge by using ‘feature toggling’, i.e. marking the feature code so it is inaccessible at run-time. On the surface, feature toggling is a relatively simple concept and easy to implement for some features (e.g. if the feature was to add a ‘SAVE’ button, we just gray it out). Problem solved, right? You can now move the whole baseline in because the new feature button is not visible so the newly developed code will not be available to the end user in production.

However, if the features had to do with complex business logic, then additional code might have been introduced marking code which should be skipped at run time. Basically, you’re creating versions (i.e. branches) of code within your embedded code base without the benefit of additional tooling to effectively support branching.

Based on our customer’s experience, feature toggling is a poor overall solution for distributed applications built using webMethods (or really any enterprise platform), since your production environment would contain even more untested code (feature + run-time versioning). When several new features and fixes are incorporated at the same time, the entire process can become completely unmaintainable.

In summary…

Be selective which webMethods projects (or really any enterprise projects) you use to leverage Continuous Integration. We’re not really pointing out anything new here other than that those same interdependency challenges can be magnified in projects that leverage multiple enterprise applications like the one above. If your project is relatively simple in architecture and functionality can be isolated to some extent, then it can add value. However, based on what we’ve seen at our customers, trying to leverage CI on many types of webMethods projects (and enterprise applications in general) may look good on paper but can be painful in practice.

At CrossVista, we support CI but generally recommend and enable the use of Scheduled Integration combined with Continuous Testing for projects that leverage webMethods and other enterprise applications. With a Scheduled Integration approach, you can get many of the promised benefits of CI (i.e. increased agility, fewer errors, less time spent merging) without the speedbumps that come with CI. We will describe how Scheduled Integration works in a subsequent blog post.

Please visit www.crossvista.com or www.scriptlessdevops.com if you would like to learn more.CrossVista provides a “Scriptless” DevOps solution for enterprise applications that manages change across the SDLC from Development through Production Deployment and everywhere in-between. The scriptless engine is integrated with technologies such as IBM IIB, MQ and WAS, SoftwareAG’s webMethods platform, Informatica’s PowerCenter, Salesforce.com, databases, file based systems, and more. The engine eliminates the need for complex scripts normally required to support these technologies… no more writing a bunch of scripts to manage your end-to-end SDLC processes.

September 1, 2016 | Scriptless DevOps Blog