Open Source Cloud Authors: Stackify Blog, Vaibhaw Pandey, Liz McMillan, Pat Romanski, Wesley Coelho

Related Topics: @CloudExpo, Microservices Expo, Open Source Cloud

@CloudExpo: Article

Talend Open Source Approach Provides Holistic Integration Capability

The role and impact of integration has shifted, and a more comprehensive and managed approach to integration is required

This discussion centers on how the role and impact of integration has shifted, and how a more comprehensive and managed approach to integration is required, thanks to such major trends as cloud, hybrid computing, and managing massive datasets.

Moreover, the tools that support enterprise integration need to be usable by more types of workers, those that are involved with business process activities and data analysis. The so-called democratization of IT effect is also rapidly progressing into this traditionally complex and isolated world of applications and data integration.

So, how do enterprises face up to the generational shift of the function of integration to new and more empowered users, so that businesses can react and exploit more applications and data resources and do so in a managed and governed fashion? This is no small task.

We're finding that modern, lightweight, and open-source platforms that leverage modular architectures are a new and proven resource for the rapid and agile integration requirements. And, the tools that support these platforms have come a long way in ease of use and applicability to more types of activities.

So we assembled a panel to discuss how these platforms have evolved, how the open-source projects are being produced and delivered into real-time and enterprise-ready, mission-critical use scenarios, and what’s now available to help make integration a core competency among more enterprise application and data activities and processes.

Please join Dan Kulp, the Vice President of Open Source Development at Talend’s Application Integration Division and also the Project Management Committee Chair of the Apache CXF Project, along with Pat Walsh, Vice President of Marketing in the Application Integration Division at Talend. The discussion is moderated by Dana Gardner, Principal Analyst at Interarbor Solutions. [Disclosure: Talend is a sponsor of BriefingsDirect podcasts.]

Here are some excerpts:

Walsh: We're seeing a couple of overriding trends that have really shifted the market for integration solutions. The needs have shifted with changes in the workplace.

First and foremost, we're seeing that there is much more information that needs to be managed, much more data associated, and there are a couple of drivers of that.

One is that there are many more interactions amongst different functional units within a business. We're seeing that silos have been broken down and that there’s more interaction amongst these different functions, and thus more data being exchanged between them and more need to integrate that data.

There’s also this notion of the consumerization of IT, that with so many devices like iPhones and iPads being accessible to consumers in their everyday life. They bring those to work and they expect those tools to be adapted to their workplace. With that just comes an even larger increase in the data explosion that you had referenced earlier.

Coupled with that are overriding trends in IT to shift the burden of supporting systems away from the traditional data center and into the cloud. Cloud has been a big movement over the last couple of years in IT and it has an impact on integration. No longer can an IT department have full control over the applications that they are integrating. They now have to interact with applications like Salesforce.com.

A number of these trends converged. In the past, you may have been able to address data issues separately with small portion of your IT group within the data center and say application integration separately with another group within the data center. Nowadays, you are not only in control of your own systems, you have to depend on systems that someone else would be supporting for you in the cloud. Thus, the complexity of all of the integration points that need to be managed has exploded.

The architectural trend is really driving the need for the data and application integration technologies and the team supporting those to come together.

These are some of the overriding trends that we are seeing at Talend and responding to in terms of issues that are driving our customer needs today.

Gardner: Why is it important for data and application integration activities to become closer or even under the same umbrella?

Walsh: The two trends that you talked about are related. The architectural trend is really driving the need for the data and application integration technologies and the team supporting those to come together. The reason is that data and application integration no longer are necessarily centralized in a single location.

When they were, you had, in essence, a single point of integration that you needed to manage amongst the data and the applications. Nowadays, it’s distributed throughout your enterprise, but also distributed, as I mentioned before, across a network of partners and providers that you may be using.

So many touch points

With that, there’s now the mandate that you can no longer isolate data from application, because the touch points are just so many. You now need to look at solutions that, from the get-go, consider both aspects of the integration problem -- the data aspect and the system and application integration aspect.

Gardner: And, I suppose we need to tool in such a way that we can approach both of these problem sets, the data integration and the applications integration, with a common interface or at least common logic. Is that correct?

Walsh: Yes, and up until now the two audiences have been treated quite differently. I think the tool expectations of the audience for data management versus the audience for application integration were quite different. We're finding that we need to bridge that gap and provide unified tool sets that are appropriate for both the data management user, as well as the application integration user.

Gardner: Why must we take a different kind of architectural step here, Dan?

Kulp: As Pat mentioned earlier, with the shifting of the requirements from silos into more of a distributed environment, the developers that are doing the application integration and the people doing the data management have to talk a lot more to get these problems solved. Your older solutions, from five years ago or whatever, that had each of those things completely separate were not able to scale up to this distributed type environment.

One aspect that open source brings is a very wide range of requirements that are placed on these open source projects. That provides a lot of benefit to an organization, as these requirements may not be required of your organization today, but you don’t really know what’s going to happen six months or a year from now.

You may acquire another company or you have to integrate another set of boxes from another area of your organization. The open source projects that you see out there, because of their open-source nature, have been attracting a wide range of developers, a wide range of new requirements and ideas, and very bright people who have really great ideas and thoughts and have made these projects very successful, just from the community nature of open source.

There is also the obvious cost benefit of not having all these high priced licenses, but the real value, in my opinion, is the community that’s behind these projects. It's continuously innovating and continuously providing new solutions for problems you may not even have yet.

Gardner: With cloud computing, you're also dealing with more moving parts. I'm quite sure that many of the cloud providers have a significant amount of open source in their infrastructure that helps make these interactions technically possible.

New complexities

Walsh: Agreed. The cloud brings a whole new set of complexities and challenges and as you are deploying your applications into the cloud, you need to think about these things. And a lot of these open-source projects that are addressing some of these cloud needs have thought about these things.

If your organization isn’t into cloud yet, but you're thinking about it, leverage the expertise that's already out there. Talk to the communities and get engaged with those communities. You'll learn a lot, and you'll be probably better off for it in the long run.

Expanded market

One interesting point to raise before talking about what we're seeing people doing is that there is an expanded market now for these integration challenges. It used to be that we would see very large enterprises were the ones that were addressing complexity in their organizations.

With cloud-based initiatives and such, it’s affecting even small to medium-size businesses (SMBs). We see a much broader set of enterprises trying to address it. Companies that have fewer than 1,000 employees are now looking at integration solutions to manage their data and their applications in the cloud in a much more sophisticated way than just three years ago. It’s a much broader problem.

The way that people are hoping to address it is by looking for a way that doesn’t require a massive outlay of investment in consulting resources. The traditional large organization, in addition to purchasing product to help them with integrating their data and integrating their applications, would typically have systems integrator help them pull everything together. That’s obviously not an affordable path for an SMB.

Therefore, people are looking to see, how they can find a combined, easy to use way and how they can gain knowledge from people who have experience, having tackled these issues and problems in the past.

We're finding that people are looking for just a simpler, prescriptive way to do the majority of the challenges out there. In terms of the 20 percent outlier problems, you may need to have a systems integrator come in and help you with that. But, people are really focused on the meat and potatoes of the integration of their functions, the data, and the applications that go along with those processes and functions.

We grab those and bring them together, the best of breed from the various Apache projects that solve real world problems.

Gardner: Five or seven years ago, this all was a very complex and costly activity. We've now been able to abstract up the value, but I also reduce and subvert the complexity. Tell me how you do that.

Kulp: The first step in that process to solve that problem was identifying where the best solutions are. They're primarily in open source. I mentioned CXF and Camel, and there is Apache Karaf providing some OSGi stuff.

That was the first step. We grab those and bring them together, the best of breed from the various Apache projects that solve real world problems.

The next step was trying to find or produce a set of tooling that makes using those products a lot easier. One of the things about Apache that you will discover, if you are heavily involved is that we are hardcore developers. For us, writing Java code to solve a problem is natural.

Skill sets

One of the problems that we're trying to address is bringing this great technology produced by the Apache people into the hands of those that don’t have that same level of skill set, expertise, or mindset.

That includes those from the application integration side, where you have developers that are used to doing point-and-click type enterprise integration pattern things, to the data integration people that are used to their data mappings, GUIs, and things like that, and trying to bring both sets of people together into a platform that can solve both teams.

Gardner: What is it about your tools and approach at Talend that is helping to bring this to the masses in a way that’s automated; a service factory approach, rather than a hand coding approach?

Walsh: Talend has a great history of unifying technologies onto a common platform, to really keep the power of the underlying tools, but simplify the interface to it. This unified platform really consists of five key components.

The first one is a common development environment that is used across the products. The second thing is a common deployment tool that allows you to deploy into a runtime environment.

By providing this unified platform of tools, it allows someone to learn a single interface, regardless of whether it’s at the development stage, the deployment stage, or the management stage.

There's also a common repository that allows you, across the lifecycle of your process, to be able to manage it consistently, regardless of the type of technology that’s being used. Finally, there is common monitoring across the entire environment.

What we are doing now is extending that model that has been applied to our data management products to encompass the ESB, the application integration aspect of it. By providing this unified platform of tools, it allows someone to learn a single interface, regardless of whether it’s at the development stage, the deployment stage, or the management stage, and get the power of master data management technologies, data integration, data quality, or the ESB technologies themselves.

By providing this one interface, this one common environment, allows people to become comfortable with this common interface, but have the benefit of multiple sets of tools.

We've gone to great lengths to include security mechanisms into the solution, so that we can have approaches whereby there are certain permissions for just individuals. Or, IT management can look at certain aspects while opening it up maybe to a broader audience, when it comes to development and use of the interfaces that are going to be developed on the data in application side.

Democratizing technology

t’s very important, as you say, that as we bring this technology to the masses, as we refer to it, democratizing the technology, lowering the barriers to entry that historically have been in place, we don’t remove any of the enterprise qualities that are expected. Security is certainly a major one, as is policy management, so that you could have a number of different business roles that allow you to have the flexibility you need as you deploy it into a large- or even medium-size enterprise.

We're providing both capabilities, simplifying the interface, while not removing any of the enterprise qualities that have come to be expected of the integration products we provide.

Gardner: Talend has also been merging and acquiring. Tell me a little bit about your business and the evolution of Talend that has allowed you to provide this all in one integration capability to the masses?

Walsh: It came quite naturally from Talend’s perspective. Data customers were using our data integration tools, as well as our data quality tools. We have Talend Open Studio, which is our popular open source data integration technology. Customers naturally were inquiring about how they could provide these data jobs as services, so that they could be reused by other applications, or they were inquiring how they could incorporate our technology into a SOA.

This led Talend to partner with a company called Sopera. They had a very rich ESB-based integration platform for applications. After two years of partnership, we decided it made sense to come together in a stronger way, and Talend acquired Sopera.

We're providing both capabilities, simplifying the interface, while not removing any of the enterprise qualities that have come to be expected of the integration products we provide.

So, we have seen this firsthand from our customers. It really drove us to see the convergence of data and application integration technology, and therefore the acquisition of Sopera’s technology, as well as the people behind that technology, has enabled us to really come in with this common platform that we are just now releasing.

We have a couple of examples that I can refer to. I think the most tangible one that may make sense to folks is that we have an insurance company that we work with. While they've been working with us for quite some time on the data side of the house, looking at how they can have their back office data shared amongst the different industry consortia that they work with to do ratings and other checks on credit worthiness or insurance risk, that has really been about integrating data on the backend.

Much like any business, they're making it more accessible to their consumers by trying to extend their back-office systems into systems that have more general web interface or maybe an interface at an ATM.

Opened to consumers

So, they required some application integration technology, and with that, they built this web interface and opened it up to consumers. The expectation of their user is a much more rapid response time. When they had to interface with an agent in the office, they may wait 24 hours for a response, but now they expect their answer to come during their web-based session.

The timeframe required has led them to have an application integration solution that can respond in sub-second response rates for their transaction. In the past, they were going with a much longer latency for the completion of transactions.

It's just a typical example that I think folks can appreciate. As people extend their back office systems to consumers, number one, consumer expectations raised the bar in terms of the overall performance of the system, and thus the technology that’s supporting those systems needs to necessarily change to support that expectation.

Gardner: In listening to Pat describe that use case, Dan, it sounds as if what we're trying to accomplish here is to do what the data warehousing, data mining, and business intelligence (BI) field have done, but perhaps allow many of those values to be extracted with more agility, faster, and then with a dynamic approach.

Is that fair? Are we really compressing or creating a category separate from BI, but that does a lot of what BI does vis-à-vis the integration of data and activities for application services?

That requires a whole new set of skills, a whole new set of challenges.

Kulp: That’s exactly what’s happening. A couple of years back, data mining ended up being batch jobs that were run at midnight or overnight. Then, the data would be available to the front end people the next morning. You'd get your reports or you'd log into your system and check the results of these batch jobs.

With extending your back-end data systems to the consumer, these overnight batch systems are really not meeting the expectations of the consumers. They're demanding that their information be available immediately. They submit a new request and they want to have things updated immediately, so that results are available and displayed within seconds, not overnight.

That requires a whole new set of skills, a whole new set of challenges. The people that were doing the front-end application integration that queried the data from the overnight batch jobs suddenly have to have some expertise in not just cleaning the data, but allowing or working with the team doing the data space, to provide updates to that information in a much more dynamic form.

Gardner: Why in the future does what we are talking about today become even more important, therefore become more critical as a core competency?

Becoming more relevant

Walsh: You can see that, as the consumerization of technology increases. We're already seeing the pressure that IT feels from becoming more relevant to the business, that just expands.

As I said before about the consumerization of devices in the workplace, it really does come down to the interfaces and the expectations that it doesn’t require a specialist in an IT field to be able to manipulate and analyze the information that they need or even to create a service or application that would enable them to do their everyday task or work function.

That’s just going to expand it. It has been happening, and we are just going to see that at a more rapid pace. It’s going to require that vendors and technology companies like Talend respond in kind and build products that are more accessible to a broader audience of users.

I think it’s analogous to what we saw in the early days of the Internet. Early on you would do command-line interfaces to send files back and forth. Once there was a web-based interface, it opened it to the masses. Nowadays, we think nothing of using a web browser to do all kinds of activity that 20 years ago was reserved to just people that had a technical know how to manipulate those systems.

We are seeing the same across these aspects of the business that up until now had really been the bastions of IT teams.

If it’s beneficial to my organization, why wouldn’t it be beneficial to others in my industry or to an even broader audience?

Today, we see that they are really addressing data services as an efficiency within their organization. How can I leverage the investment that I have made in this initial data analysis or data job across the entirety of my organization? But it’s not a big step to take beyond that to say, if it’s beneficial to my organization, why wouldn’t it be beneficial to others in my industry or to an even broader audience?

So we absolutely see that as a level of commerce that will be enabled by more sophisticated data services, technology, with a more accessible interface to that technology.

Comes down to consumers

Kulp: It really comes down to the consumers of these services and data. As the markets have expanded and the consumers are demanding things to get their information faster or get more information or advertisers need to figure out, where are these consumers going and just the whole variety of information sources expand out as well, the architecture of the applications and the interactions between the front end and backend systems kind of get blurred.

Things are changing, and companies like Talend that are involved in the space need to adapt as well and provide better solutions that make these blurring lines occur a lot quicker. That’s what we are trying to target today.

You may also be interested in:

More Stories By Dana Gardner

At Interarbor Solutions, we create the analysis and in-depth podcasts on enterprise software and cloud trends that help fuel the social media revolution. As a veteran IT analyst, Dana Gardner moderates discussions and interviews get to the meat of the hottest technology topics. We define and forecast the business productivity effects of enterprise infrastructure, SOA and cloud advances. Our social media vehicles become conversational platforms, powerfully distributed via the BriefingsDirect Network of online media partners like ZDNet and IT-Director.com. As founder and principal analyst at Interarbor Solutions, Dana Gardner created BriefingsDirect to give online readers and listeners in-depth and direct access to the brightest thought leaders on IT. Our twice-monthly BriefingsDirect Analyst Insights Edition podcasts examine the latest IT news with a panel of analysts and guests. Our sponsored discussions provide a unique, deep-dive focus on specific industry problems and the latest solutions. This podcast equivalent of an analyst briefing session -- made available as a podcast/transcript/blog to any interested viewer and search engine seeker -- breaks the mold on closed knowledge. These informational podcasts jump-start conversational evangelism, drive traffic to lead generation campaigns, and produce strong SEO returns. Interarbor Solutions provides fresh and creative thinking on IT, SOA, cloud and social media strategies based on the power of thoughtful content, made freely and easily available to proactive seekers of insights and information. As a result, marketers and branding professionals can communicate inexpensively with self-qualifiying readers/listeners in discreet market segments. BriefingsDirect podcasts hosted by Dana Gardner: Full turnkey planning, moderatiing, producing, hosting, and distribution via blogs and IT media partners of essential IT knowledge and understanding.

@ThingsExpo Stories
In his session at 21st Cloud Expo, Carl J. Levine, Senior Technical Evangelist for NS1, will objectively discuss how DNS is used to solve Digital Transformation challenges in large SaaS applications, CDNs, AdTech platforms, and other demanding use cases. Carl J. Levine is the Senior Technical Evangelist for NS1. A veteran of the Internet Infrastructure space, he has over a decade of experience with startups, networking protocols and Internet infrastructure, combined with the unique ability to it...
"There's plenty of bandwidth out there but it's never in the right place. So what Cedexis does is uses data to work out the best pathways to get data from the origin to the person who wants to get it," explained Simon Jones, Evangelist and Head of Marketing at Cedexis, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
"Cloud Academy is an enterprise training platform for the cloud, specifically public clouds. We offer guided learning experiences on AWS, Azure, Google Cloud and all the surrounding methodologies and technologies that you need to know and your teams need to know in order to leverage the full benefits of the cloud," explained Alex Brower, VP of Marketing at Cloud Academy, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clar...
Large industrial manufacturing organizations are adopting the agile principles of cloud software companies. The industrial manufacturing development process has not scaled over time. Now that design CAD teams are geographically distributed, centralizing their work is key. With large multi-gigabyte projects, outdated tools have stifled industrial team agility, time-to-market milestones, and impacted P&L stakeholders.
Gemini is Yahoo’s native and search advertising platform. To ensure the quality of a complex distributed system that spans multiple products and components and across various desktop websites and mobile app and web experiences – both Yahoo owned and operated and third-party syndication (supply), with complex interaction with more than a billion users and numerous advertisers globally (demand) – it becomes imperative to automate a set of end-to-end tests 24x7 to detect bugs and regression. In th...
"Akvelon is a software development company and we also provide consultancy services to folks who are looking to scale or accelerate their engineering roadmaps," explained Jeremiah Mothersell, Marketing Manager at Akvelon, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
"MobiDev is a software development company and we do complex, custom software development for everybody from entrepreneurs to large enterprises," explained Alan Winters, U.S. Head of Business Development at MobiDev, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
SYS-CON Events announced today that CrowdReviews.com has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5–7, 2018, at the Javits Center in New York City, NY. CrowdReviews.com is a transparent online platform for determining which products and services are the best based on the opinion of the crowd. The crowd consists of Internet users that have experienced products and services first-hand and have an interest in letting other potential buye...
"IBM is really all in on blockchain. We take a look at sort of the history of blockchain ledger technologies. It started out with bitcoin, Ethereum, and IBM evaluated these particular blockchain technologies and found they were anonymous and permissionless and that many companies were looking for permissioned blockchain," stated René Bostic, Technical VP of the IBM Cloud Unit in North America, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Conventi...
SYS-CON Events announced today that Telecom Reseller has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5-7, 2018, at the Javits Center in New York, NY. Telecom Reseller reports on Unified Communications, UCaaS, BPaaS for enterprise and SMBs. They report extensively on both customer premises based solutions such as IP-PBX as well as cloud based and hosted platforms.
"Space Monkey by Vivent Smart Home is a product that is a distributed cloud-based edge storage network. Vivent Smart Home, our parent company, is a smart home provider that places a lot of hard drives across homes in North America," explained JT Olds, Director of Engineering, and Brandon Crowfeather, Product Manager, at Vivint Smart Home, in this SYS-CON.tv interview at @ThingsExpo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
Coca-Cola’s Google powered digital signage system lays the groundwork for a more valuable connection between Coke and its customers. Digital signs pair software with high-resolution displays so that a message can be changed instantly based on what the operator wants to communicate or sell. In their Day 3 Keynote at 21st Cloud Expo, Greg Chambers, Global Group Director, Digital Innovation, Coca-Cola, and Vidya Nagarajan, a Senior Product Manager at Google, discussed how from store operations and ...
It is of utmost importance for the future success of WebRTC to ensure that interoperability is operational between web browsers and any WebRTC-compliant client. To be guaranteed as operational and effective, interoperability must be tested extensively by establishing WebRTC data and media connections between different web browsers running on different devices and operating systems. In his session at WebRTC Summit at @ThingsExpo, Dr. Alex Gouaillard, CEO and Founder of CoSMo Software, presented ...
WebRTC is great technology to build your own communication tools. It will be even more exciting experience it with advanced devices, such as a 360 Camera, 360 microphone, and a depth sensor camera. In his session at @ThingsExpo, Masashi Ganeko, a manager at INFOCOM Corporation, introduced two experimental projects from his team and what they learned from them. "Shotoku Tamago" uses the robot audition software HARK to track speakers in 360 video of a remote party. "Virtual Teleport" uses a multip...
A strange thing is happening along the way to the Internet of Things, namely far too many devices to work with and manage. It has become clear that we'll need much higher efficiency user experiences that can allow us to more easily and scalably work with the thousands of devices that will soon be in each of our lives. Enter the conversational interface revolution, combining bots we can literally talk with, gesture to, and even direct with our thoughts, with embedded artificial intelligence, whic...
SYS-CON Events announced today that Evatronix will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Evatronix SA offers comprehensive solutions in the design and implementation of electronic systems, in CAD / CAM deployment, and also is a designer and manufacturer of advanced 3D scanners for professional applications.
Leading companies, from the Global Fortune 500 to the smallest companies, are adopting hybrid cloud as the path to business advantage. Hybrid cloud depends on cloud services and on-premises infrastructure working in unison. Successful implementations require new levels of data mobility, enabled by an automated and seamless flow across on-premises and cloud resources. In his general session at 21st Cloud Expo, Greg Tevis, an IBM Storage Software Technical Strategist and Customer Solution Architec...
To get the most out of their data, successful companies are not focusing on queries and data lakes, they are actively integrating analytics into their operations with a data-first application development approach. Real-time adjustments to improve revenues, reduce costs, or mitigate risk rely on applications that minimize latency on a variety of data sources. In his session at @BigDataExpo, Jack Norris, Senior Vice President, Data and Applications at MapR Technologies, reviewed best practices to ...
An increasing number of companies are creating products that combine data with analytical capabilities. Running interactive queries on Big Data requires complex architectures to store and query data effectively, typically involving data streams, an choosing efficient file format/database and multiple independent systems that are tied together through custom-engineered pipelines. In his session at @BigDataExpo at @ThingsExpo, Tomer Levi, a senior software engineer at Intel’s Advanced Analytics gr...
When talking IoT we often focus on the devices, the sensors, the hardware itself. The new smart appliances, the new smart or self-driving cars (which are amalgamations of many ‘things’). When we are looking at the world of IoT, we should take a step back, look at the big picture. What value are these devices providing? IoT is not about the devices, it’s about the data consumed and generated. The devices are tools, mechanisms, conduits. In his session at Internet of Things at Cloud Expo | DXWor...