The future of Extensibility for Automation Integration

What is Automation Extensibility?

No automation software will accomplish 100% of your requirements out of the box.  Often, I am asked, “Can you automate my application?” This is probably not the best question to ask. Automation engineers know the answer to that question is always “yes”. We are always figuring out innovative ways to make things work together.  The better question is “What is the effort required to integrate to my application?”. This question often gives us pause and makes us dig deeper in our understanding.  We have peeled back the layers and think about some of the details of the integration. What are the performance requirements? How will I manage and monitor this? What will the payload look like?  What APIs are available? What language will I use? How will I secure it? What will this cost to run? And perhaps most importantly, what platform will I use to accomplish the integration that will satisfy all the requirements?

Luckily, if you are running vSphere, you already have a giant head-start to answer this question regardless of your technology preferences.

“Old School”: Oldies but goodies

Traditionally, most automation in a vSphere environment uses vRealize Orchestrator(vRO) for extensibility.  vRO gives you the ability to write javascript workflows to leverage out-of-the-box plug-ins for many COTS applications as well as built-in support for HTTPS, SOAP, REST, SSH, and PowerShell integration work.  This provides a good foundation for most extensibility use-cases.

If you also have vRealize Automation, you get a “webhooks-like” interface called the Event Broker built on-top of vRO enabling you to trigger workflow execution with relevant payloads based on events within your provisioning lifecycle.  These are powerful tools, but it has some drawbacks based on modern application architectures.

There are many good reasons to use vRealize Orchestrator.  It gives you complete control over the performance and scalability of the underlying infrastructure.  It is extremely secure and provides full multi-tenancy and RBAC access to all components. It has a rich ecosystem with large array of out-of-the-box integrations already built, tested, and supported.  Giving you the ability to focus efforts on the mission critical components.

However, vRealize Automation can be a heavy lift , especially for emerging organizations.  There is a cost in both personnel and infrastructure maintaining the environment .  This can increase the cost of entry and time to value when considering it against a SaaS platform with Webhooks.

Additionally, there can be a steep learning curve with vRealize Orchestrator.  It is common for powerful solutions to require a fair amount of learning.  However vRO can be especially steep because everything is built-in, including the IDE, data-structures, debugger, editor, etc.  It does not allow you to leverage components that you may already be familiar with to lessen the learning requirements. Another side-effect of this all-in-one architecture of vRO is that it does not play well with current development methodologies that include external source control and everything-as-code approaches.  All vRO artifacts must be exported to allow for pipeline integration, and even then, some of the workflow components are based on visualization technologies and not easily consumable as YAML or it’s like.

vRealize Automation with vRealize Orchestrator is a great solution if you are looking for a mature and proven automation platform where you have control of the infrastructure to ensure availability and performance.  It also shines if you want to leverage a lot of existing integrations to reduce the technical debt in your organization with out-of-the-box plug-ins.

New Kids on the Block

So, what are your options if you are looking for a “lighter” automation solution?

Two “newer” automation approaches are webhooks and serverless extensibility.  These technologies have gotten a lot of exposure in the last few years and are continuing to gain momentum in the enterprise automation space.  While there are some significant benefits to using them, they also have some drawbacks that need to be considered when selecting a platform for an automation integration project.

Webhooks

A webhook is a HTTP callback that asynchronously issues a POST with a pre-defined payload in response to an application event.  You, as the integrator, are responsible for creating an application that receives and parses the information to do something interesting.

Webhooks offer a lot of power with a low cost of entry.  Since they are asynchronous, you don’t have to create and support any type of polling mechanism.  This simplifies most scaling and race condition issues. It also eliminates most of the requirements around stateful transaction tracking because that information is usually part of the payload generated from the webhook and can usually be processed statelessly.

Since the payload is usually JSON or XML-based that you are responsible for parsing, you are free to choose whatever language and tools that you want to use.  This greatly reduces the overall learning curve. However, there are some drawbacks to webhooks that you should be aware of.

  • You need to have a good idea of the possible volume of events that could be triggered and ensure that your backend automation can handle the load.
  • This raises another issue with webhooks.  How do you monitor them? You typically do not have any insight into how the webhooks are behaving.  Are they down? Is there a run-away process? If your application is mission-critical and you are relying on webhooks, you will need to develop some monitoring strategy to ensure you do not have a failure at a critical time.
  • Since webhooks are asynchronous and event-driven, testing and troubleshooting them can also be more complicated.  You need to ensure you can account for all possible payload values, even ones you may not have expected.
  • Finally, security can be an issue with webhooks.  You need to ensure that data being passed is fully encrypted and appropriate.  You will also need to ensure that your back-end is not vulnerable to spoofing and man-in-the-middle attacks.

 

Serverless- Functions as a Service (FaaS)

Webhook’s emerging big-brother in the automation arena is ‘serverless’ or Functions as a Service (FaaS). Some common examples of FaaS serverless technologies are AWS Lambda and OpenFaaS. Of course, everything still runs on servers, but “serverless” means the automation developer does not have to worry about the infrastructure.  From the developer point of view, it eliminates things like server provisioning, application scaling, Operating System nuances, etc. The automation developer can focus on the code and the integration.

This has some significant benefits to the organization:

  • Most serverless technologies are designed to handle auto-scaling for load automatically. You don’t have to code differently for high demand.  The infrastructure will automatically horizontally scale to meet your demand.
  • You don’t have to worry about server sizing or under-utilized resources.  Typically, serverless is a Pay-as-you go model and you will only be charged based on how much your code runs, not for any excess server capacity.
  • Serverless is designed to be vendor-agnostic.  Since you are not tied to any specific OS or platform, you can re-platform your application without significant modification.
  • Serverless is designed to integrate into modern application development and release pipelines because it is based on the “everything as code” model.  It is easy to version control each update with a source-code repository like Git, and model and deploy your integration with your application stack as part of the release process.

These features provide a lot of power and flexibility to the automation developer by reducing friction and increasing the development speed of any integration project.  However, this does come with a cost.

  • Like webhooks, you do not typically have much insight into how your code is performing or what resources are driving the auto-scaling.  It is possible that your code may be resource starved and the infrastructure is not scaling correctly. How would you know?
  • With serverless, you may find that you have code scattered in many different places.  Managing code across multiple platforms can soon turn into a nightmare and could introduce new bugs or even worse, rogue code that a developer has put up for a “quick fix” that is forgotten about until it breaks.
  • The learning curve for serverless is much steeper than that of webhooks and you often cannot leverage existing skill sets because the platform will determine the tools and languages you must use.
  • Finally, security is still an issue.  You may be storing proprietary code and IP on external sources.  You will need a mature security model to handle multi-tenancy, RBAC, and code isolation for your applications.   This is in addition to all the normal data protection and threat vulnerability work you need to do.

Musical Chairs – VMWare Dispatch

There are lots of options when choosing a platform for integration projects.  You need to understand the benefits and drawbacks for each approach and choose the one that best fits your requirements.

However, VMware is working to bridge the best aspects of each of these approaches and eliminate the drawbacks with a new open source project called Dispatch. Dispatch is not another FaaS offering.  It is designed to allow enterprises to build and manage applications and automation extensibility by adding enterprise-class features on top of an existing FaaS layer. It has current support for OpenFaaS and Riff with future support for AWS Lambda and others.

  • Just like Kubernetes brings additional value to a container-based infrastructure by adding needed features, Dispatch builds on the strengths of a serverless architecture by adding features like RBAC support, Multi-tenancy, and a rich API.
  • Because Dispatch can be deployed both on-prem or in the cloud, you get to choose the control you need for your data and applications.  This eliminates some of the data sovereignty and control issues you have with traditional serverless platforms.
  • Security is built-in to the framework to allow full RBAC control based on OAuth 2.0 backed by GitHub or Active Directory.
  • Code is kept updated with direct integration into your corporate source code repository, ensuring all your artifacts and code remain under a centralized management structure.
  • Dispatch’s API built on the Kong Microservice API Gateway that enables you to build webhooks as needed to trigger functions in support of your automation extensibility project.  This allows you to use the right tool for the right job without sacrificing capability, manageability, or security.
  • Finally, you can monitor the dispatch infrastructure with your existing toolset to giving you visibility into all aspects of your automation pipeline and ensuring appropriate availability and performance of your mission critical applications.

Summary

The future of automation extensibility is bright.  When you must determine the level of effort for an automation project, there are a lot of good options, but there can also be lot of questions.  What are the performance requirements? How will I manage and monitor this? What will the payload look like? What APIs are available? What language will I use?  How will I secure it? What will this cost to run?

VMware Dispatch offers the most options with the best security and serviceability model for enterprise-class applications. It gives you a best in breed extensibility platform for automation that combines the best of traditional and emerging technologies to meet these requirements and more.

For more information on VMware Dispatch:
https://vmware.github.io/dispatch/
https://github.com/vmware/dispatch
https://vmware.github.io/dispatch/documentation/front/overview
https://blogs.vmware.com/opensource/2018/01/12/dispatch-project-open-source-serverless-framework/