Welcome!

Open Source Cloud Authors: Liz McMillan, Elizabeth White, Pat Romanski, Jeev Trika, Ed Featherston

Related Topics: @DevOpsSummit, Java IoT, Microservices Expo, Linux Containers, @CloudExpo, @BigDataExpo

@DevOpsSummit: Article

The DevOps Database | Part 3

Applying Feedback Loops to Database Change Management

In the third post in this series, I’d like to talk about the Second Way of DevOps: Amplifying Feedback Loops.  Here’s a refresher on The Second Way from my introductory post in this series:

The Second Way: Amplify Feedback Loops – This Way deals primarily with facilitating easier and faster communication between all individuals in a DevOps organization.  The goals of this step are to foster better understanding of all internal and external customers in the process and to develop an accessible body of knowledge to replace the dependence on expertise scattered across individuals.

I’ve stated before in this series that Database Change Management poses a unique challenge when your organization is shifting to an agile development methodology and implementing DevOps patterns.  Unlike other areas of your application stack, responsibility for managing application schema straddles two groups operating under somewhat opposed expectations. The development group is on the hook for producing more and more business critical features and releases at an ever increasing rate.  DBAs are tasked with providing a secure, highly available data platform and protecting the integrity of the organization’s priceless data.  The rate of schema change required by development to satisfy expectations can run head long into a database change process that is deliberate and metered by necessity to avoid downtime and data loss.  In organizations where these two groups are isolated from each other, you have the makings of a bottle neck in your release process.

The solution to this problem is embodied by The Second Way of DevOps. Communicate early, communicate often, communicate broadly, and prepare for what’s ahead. The tricky part is implementing the solution in a way that’s meaningful to every stakeholder in an organization’s application group.  At Datical, we’ve spent just as much time on how we organize and present the data associated with application schema changes as we have on automating the deployment of these changes.  We’ve rallied around the following key concepts to bring the The Second Way of DevOps to Database Change Management.

Proactive, Predictive Change Analysis
In an organization where development works independently of the database group, truly understanding the impact a stack of SQL scripts will have on downstream environments is a tedious and time consuming task.  Before these changes can be promoted, target environments must be meticulously evaluated for conflicts and dependencies that will impact the deployment process.  This often involves manual reviews and comparisons of diagrams and database dumps of complex environments. Achieving a high degree of confidence in the success of the proposed updates is difficult because it is so easy to overlook something.  Datical has developed a patent pending simulation feature called Forecast that automates this process.  The Forecast feature builds an in memory model of the target environment, simulates proposed changes on top of that model, and warns of potential error conditions, data loss and performance issues without touching the target database.  Because there is no impact to target environments, database administrators can Forecast changes several times during the development cycle to get ahead of issues that would normally be discovered much later in a pre-release review.  Development gets regular feedback on the changes they are proposing and can address issues that arise during the initial development phase when it is easier and safer to resolve them.   The two teams are working in unison to ensure a safe database deployment that works the first time without surprises.

Always Remember Where You Came From
Database changes are usually designed to address the immediate goals of an organization.  Once one set of requirements has been satisfied by a release, the motivations for the design decisions made for that release generally fades away as new requirements come along and new business initiatives take center stage.  Comments in SQL scripts and on the database objects themselves can be helpful in determining why things are the way they are, but these traces of the past are scattered everywhere. Making sense of the whole is an exercise in archaeology.   This was one of the driving forces behind our model based approach to database change management.  Our model is architected to provide a living history of your application schema.  Individual changes are tied to the specific requirement and release that necessitated them.  This data lives in the model so the information you need to make intelligent design decisions is right in front of you when you need it.

Know Where You Are
By tying the business reasons behind each schema change in the model, this information can be tracked in each database instance as it’s updated and included in Forecast, Deploy, and historical reports.  Tracking the changes in each instance and providing detailed reports allows you to easily disseminate information, effectively gate deployment steps, and quickly satisfy audit requirements. When everyone in your organization has access to thorough accounts of the Who, What, Where, When, and Why of any single database change in any environment, everyone is operating on the same level and can more effectively work towards a common goal.

Know Where You’re Headed
The model also facilitates concurrent development on multiple releases of a project.  By tracking changes made for several different releases in a single model, the development teams working on these releases are able to collaborate and stay ahead of changes made by other teams that may impact future releases.  Developers are able to unify redundant changes and eliminate conflicting changes as they implement instead of spending time on redesign later in the process when time is scarce and the cost of change is high.

More Stories By Pete Pickerill

Pete Pickerill is Vice President of Products and Co-founder of Datical. Pete is a software industry veteran who has built his career in Austin’s technology sector. Prior to co-founding Datical, he was employee number one at Phurnace Software and helped lead the company to a high profile acquisition by BMC Software, Inc. Pete has spent the majority of his career in successful startups and the companies that acquired them including Loop One (acquired by NeoPost Solutions), WholeSecurity (acquired by Symantec, Inc.) and Phurnace Software.

@ThingsExpo Stories
There will be new vendors providing applications, middleware, and connected devices to support the thriving IoT ecosystem. This essentially means that electronic device manufacturers will also be in the software business. Many will be new to building embedded software or robust software. This creates an increased importance on software quality, particularly within the Industrial Internet of Things where business-critical applications are becoming dependent on products controlled by software. Qua...
"There's a growing demand from users for things to be faster. When you think about all the transactions or interactions users will have with your product and everything that is between those transactions and interactions - what drives us at Catchpoint Systems is the idea to measure that and to analyze it," explained Leo Vasiliou, Director of Web Performance Engineering at Catchpoint Systems, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York Ci...
SYS-CON Events has announced today that Roger Strukhoff has been named conference chair of Cloud Expo and @ThingsExpo 2016 Silicon Valley. The 19th Cloud Expo and 6th @ThingsExpo will take place on November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. "The Internet of Things brings trillions of dollars of opportunity to developers and enterprise IT, no matter how you measure it," stated Roger Strukhoff. "More importantly, it leverages the power of devices and the Interne...
The Internet of Things will challenge the status quo of how IT and development organizations operate. Or will it? Certainly the fog layer of IoT requires special insights about data ontology, security and transactional integrity. But the developmental challenges are the same: People, Process and Platform. In his session at @ThingsExpo, Craig Sproule, CEO of Metavine, demonstrated how to move beyond today's coding paradigm and shared the must-have mindsets for removing complexity from the develo...
SYS-CON Events announced today that MangoApps will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. MangoApps provides modern company intranets and team collaboration software, allowing workers to stay connected and productive from anywhere in the world and from any device.
The IETF draft standard for M2M certificates is a security solution specifically designed for the demanding needs of IoT/M2M applications. In his session at @ThingsExpo, Brian Romansky, VP of Strategic Technology at TrustPoint Innovation, explained how M2M certificates can efficiently enable confidentiality, integrity, and authenticity on highly constrained devices.
"We've discovered that after shows 80% if leads that people get, 80% of the conversations end up on the show floor, meaning people forget about it, people forget who they talk to, people forget that there are actual business opportunities to be had here so we try to help out and keep the conversations going," explained Jeff Mesnik, Founder and President of ContentMX, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York City, NY.
Internet of @ThingsExpo has announced today that Chris Matthieu has been named tech chair of Internet of @ThingsExpo 2016 Silicon Valley. The 6thInternet of @ThingsExpo will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA.
When people aren’t talking about VMs and containers, they’re talking about serverless architecture. Serverless is about no maintenance. It means you are not worried about low-level infrastructural and operational details. An event-driven serverless platform is a great use case for IoT. In his session at @ThingsExpo, Animesh Singh, an STSM and Lead for IBM Cloud Platform and Infrastructure, will detail how to build a distributed serverless, polyglot, microservices framework using open source tec...
The 19th International Cloud Expo has announced that its Call for Papers is open. Cloud Expo, to be held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA, brings together Cloud Computing, Big Data, Internet of Things, DevOps, Digital Transformation, Microservices and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding business opportuni...
From wearable activity trackers to fantasy e-sports, data and technology are transforming the way athletes train for the game and fans engage with their teams. In his session at @ThingsExpo, will present key data findings from leading sports organizations San Francisco 49ers, Orlando Magic NBA team. By utilizing data analytics these sports orgs have recognized new revenue streams, doubled its fan base and streamlined costs at its stadiums. John Paul is the CEO and Founder of VenueNext. Prior ...
A critical component of any IoT project is what to do with all the data being generated. This data needs to be captured, processed, structured, and stored in a way to facilitate different kinds of queries. Traditional data warehouse and analytical systems are mature technologies that can be used to handle certain kinds of queries, but they are not always well suited to many problems, particularly when there is a need for real-time insights.
CenturyLink has announced that application server solutions from GENBAND are now available as part of CenturyLink’s Networx contracts. The General Services Administration (GSA)’s Networx program includes the largest telecommunications contract vehicles ever awarded by the federal government. CenturyLink recently secured an extension through spring 2020 of its offerings available to federal government agencies via GSA’s Networx Universal and Enterprise contracts. GENBAND’s EXPERiUS™ Application...
"My role is working with customers, helping them go through this digital transformation. I spend a lot of time talking to banks, big industries, manufacturers working through how they are integrating and transforming their IT platforms and moving them forward," explained William Morrish, General Manager Product Sales at Interoute, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York City, NY.
Big Data engines are powering a lot of service businesses right now. Data is collected from users from wearable technologies, web behaviors, purchase behavior as well as several arbitrary data points we’d never think of. The demand for faster and bigger engines to crunch and serve up the data to services is growing exponentially. You see a LOT of correlation between “Cloud” and “Big Data” but on Big Data and “Hybrid,” where hybrid hosting is the sanest approach to the Big Data Infrastructure pro...
Internet of @ThingsExpo, taking place November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with the 19th International Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world and ThingsExpo Silicon Valley Call for Papers is now open.
The IoT is changing the way enterprises conduct business. In his session at @ThingsExpo, Eric Hoffman, Vice President at EastBanc Technologies, discussed how businesses can gain an edge over competitors by empowering consumers to take control through IoT. He cited examples such as a Washington, D.C.-based sports club that leveraged IoT and the cloud to develop a comprehensive booking system. He also highlighted how IoT can revitalize and restore outdated business models, making them profitable ...
We all know the latest numbers: Gartner, Inc. forecasts that 6.4 billion connected things will be in use worldwide in 2016, up 30 percent from last year, and will reach 20.8 billion by 2020. We're rapidly approaching a data production of 40 zettabytes a day – more than we can every physically store, and exabytes and yottabytes are just around the corner. For many that’s a good sign, as data has been proven to equal money – IF it’s ingested, integrated, and analyzed fast enough. Without real-ti...
I wanted to gather all of my Internet of Things (IOT) blogs into a single blog (that I could later use with my University of San Francisco (USF) Big Data “MBA” course). However as I started to pull these blogs together, I realized that my IOT discussion lacked a vision; it lacked an end point towards which an organization could drive their IOT envisioning, proof of value, app dev, data engineering and data science efforts. And I think that the IOT end point is really quite simple…
With 15% of enterprises adopting a hybrid IT strategy, you need to set a plan to integrate hybrid cloud throughout your infrastructure. In his session at 18th Cloud Expo, Steven Dreher, Director of Solutions Architecture at Green House Data, discussed how to plan for shifting resource requirements, overcome challenges, and implement hybrid IT alongside your existing data center assets. Highlights included anticipating workload, cost and resource calculations, integrating services on both sides...