Welcome!

Open Source Cloud Authors: Pat Romanski, Liz McMillan, Stackify Blog, Wesley Coelho, Rainer Ersch

Related Topics: Cloud Security, Java IoT, Microservices Expo, Linux Containers, Open Source Cloud, Containers Expo Blog

Cloud Security: Article

The Definitive Guide to Heartbleed and Enterprise Mobility

There has been a lot of media attention on Heartbleed and as always that means a lot of bizarre and often conflicting advice

By Peter Rogers, Principal Mobility Architect, Cognizant

There has been a lot of media attention on Heartbleed and as always that means a lot of bizarre and often conflicting advice.  I sat down (for a very long time) to find the truth of the matter and bring it together in one single article for those involved in enterprise mobility.

What is Heartbleed?
"The [Heartbleed] bug is in the OpenSSL's implementation of the TLS/DTLS (transport layer security protocols) heartbeat extension (RFC6520). When it is exploited it leads to the leak of memory contents from the server to the client, and from the client to the server. The bug was introduced to OpenSSL in December 2011 and has been out in the wild since OpenSSL release 1.0.1 on 14th of March 2012. OpenSSL 1.0.1g released on 7th of April 2014 fixes the bug."
[http://www.heartbeat.com/]

"An attacker can retrieve a block of memory of the server up to 64kb. There is no limit on the number of attacks that can be performed. The attacker has no control over the memory region where the block is read from. Sensitive information that can be obtained is: SSL private keys; Basic authorization strings (username / password combinations); and source code [most important also cookies]. This bug affects both sides of the connection. Not only will client certificates not save you from having to update your server certificate, they can be read from the client (along with your username, password etc.) by any server you connect to."
[http://blog.fox-it.com/2014/04/08/openssl-heartbleed-bug-live-blog/]

"The Heartbleed bug is an example of Buffer Over-Read and that means that the offending OpenSSL heartbeat code does not check that this length is the actual length sent in the heartbeat request, allowing the request to ask for more data than it should be able to retrieve. The code then copies the amount of data indicated by the length from the incoming message to the outgoing message. If the length is longer than the incoming message, the software just keeps copying data past the end of the message. Since the length variable is 16 bits, you can request up to 65,535 bytes from memory. The data that lives past the end of the incoming message is from a kind of no-man's land that the program should not be accessing and may contain data left behind from other parts of OpenSSL."
[http://blog.cloudflare.com/answering-the-critical-question-can-you-get-private-ssl-keys-using-heartbleed]

The non-technical explanation from XKCD is slightly easier to follow (http://xkcd.com/1354/).

Here is a very good FAQ and an actual screen shot of a Heartbleed attack from Fox-IT.

The important thing to note is that not only can a client potentially read data from a server but the converse is also true. This impacts mobile clients too and although Apple claim they are safe there were allegedly concerns around Android 4.1.1. If a client application is vulnerable on a mobile or desktop device then equally the server can act as a honey trap to steal information from the client. Web browser software and IM clients are the biggest concern, so please make sure you don't use obscure and vulnerable software.

Who is stealing what and why?

The important question is how we got into this mess. The first thing to point out is that this is not a floor in SSL/TSP, instead it is an implementation problem in the open source OpenSSL library.  A major contributing factor has been that TLS versions 1.2 came with OpenSSL 1.0.1 and after the BEAST attack everyone rushed to implement TLS 1.2. The current theory doing the rounds is that the NSA had already ‘found ways around' SSL some time ago and this could have been by taking advantage of the Heartbleed bug. It is however noted that Man in the Middle (MITM) techniques are far easier, especially as some large companies act as their own Certificate Authority (CA). Edward Snowden leaked the existence of BULLRUN (a highly classified decryption program run by the NSA) and subsequently Edgehill (a similar program run by GCHQ) which subsequently prompted Google to double the size of their encryptions keys. We know that a number of companies knew about the vulnerability before it went public because they stated they had applied the patches at least a week ago. If you think about it a minute then OpenSSL is open source code which meant anybody could have worked it out and kept it a secret.

They say the exploit leaves no trace and so nobody actually knows if they have been targeted. Actually that statement isn't strictly true. A single heartbeat can potentially grab 64K of memory however an attacker can either keep reconnecting or during an active TLS connection keep requesting arbitrary number of 64 kilobyte chunks of memory content. There are now companies who are configuring their network monitoring software to watch for abnormal behaviour such as Fox-IT, who have developed a set of Snort signatures to detect successful exploitation of the bug.

The most important thing is what happens next and most hackers will be after what Heartbleed.com classes as primary key material.

"These are the crown jewels, the encryption keys themselves. Leaked secret keys allows the attacker to decrypt any past and future traffic to the protected services and to impersonate the service at will. Any protection given by the encryption and the signatures in the X.509 certificates can be bypassed. Recovery from this leak requires patching the vulnerability, revocation of the compromised keys and reissuing and redistributing new keys. Even doing all this will still leave any traffic intercepted by the attacker in the past still vulnerable to decryption. All this has to be done by the owners of the services."
[http://www.heartbeat.com/]

Let's imagine that a service provider has been using a vulnerable version of Open SSL 1.0.1 and the types of hacker that may target them based on time ranges:

  1. Soon after the vulnerability was released (14th of March 2012)
  2. Two or three weeks before the announcement was made public but was still being discussed in hushed circles
  3. Ever since the announcement (April 7th)
  4. Ever since the Heartbleed proof of concepts went live (April 8th)

The first group are likely to be very serious hackers or even possibly Government sponsored activities which are nothing to worry about unless you are of significant danger to anyone. The second group are likely to be serious hackers or service providers testing out vulnerabilities in their systems (and potentially partner solutions) after they learn of the issues. The third group is the rest of the world and depends on the general availability of OpenSSL vulnerability scanning software at that date. The fourth group covers hobbyist hackers because the proof of concepts are open source and highly available.

Unfortunately as of April 8th then anyone in the world can now test for this vulnerability using this widely shared Python Script (https://gist.github.com/takeshixx/10107280) and that puts it in the domain of hobbyists. Anyone can now dump a bit of RAM from a vulnerable server. It would definitely take a professional who can write their own script to make repeated heartbeats attacks and then got lucky with the timing of a server restart on a vulnerable operating system to be able to pull out a private encryption key. However, even an amateur can use Session Hijacking which means you look for cookies containing session IDs. In a single grab of memory then you are equally as likely to obtain a session ID or a username and password combination, and this is slightly harder to track for abnormal behaviour.

An example of which is running the Python proof of concept against a vulnerable JIRA ticketing system and pulling out a JSESSIONID (which is JIRA's way of tracking your HTTP session). If the system requires authentication then you can just insert the stolen cookie into the browser and become that user on the JIRA installation. (https://www.mattslifebytes.com/?p=533)

Can you actually steal a private key?

The Heartbleed bug is an example of buffer over-read and here is the offending code for the programmers amongst us:

p = &s->s3->rrec.data[0]

[...]

hbtype = *p++;
n2s(p, payload);
pl = p;

[...]

buffer = OPENSSL_malloc(1 + 2 + payload + padding);
bp = buffer;

[...]

memcpy(bp, pl, payload);

In NGINX, the keys are loaded immediately when the process is started, which puts the keys very low in the memory space. This makes it unlikely that incoming requests will be allocated with a lower address space. If NGINX is reloaded, it starts a new process and loads the keys right away, putting them at a low address. On NGINX you can therefore retrieve user credentials, cookies and even public parts of the certificate but so far not the private part. It all depends therefore on the operating system and how often the servers are restarted. CloudFlare have even issued a challenge by leaving a server running nginx-1.5.13 linked against OpenSSL 1.0.1.f on Ubuntu 13.10 x86_6

That said there appears to be a proof of concept working on Apache for a first request which corresponds with the server being rebooted.

The Certificate Revocation Tsunami
There is a genuine importance to the CloudFlare challenge and that is that if private keys can be stolen then there will be a whole lot of companies requesting certificates all at once leading to a so called Certificate Revocation Tsunami. Certificate Authorities are supposed to revoke certificates within 24 hours if there is any evidence of a key compromise. However, even if all of the affected certificates were to be revoked, contemporary web browser software handles certificate revocation poorly. For example, some browsers only perform OCSP revocation checks for Extended Validation certificates, while others ignore certificate revocation lists completely. The other concern is the rush to create new certificates will lead to incorrect certificates being generated which in turn leads to Man In The Middle (MITM) exploits down the line.

"The certificate authority infrastructure was never built to do a mass revocation of this many certificates and because of the way the infrastructure is built, if you did do a mass revocation of millions of certificates it would significantly slow down the performance of the Internet itself, which is potentially very, very bad."
[http://www.washingtonpost.com/blogs/the-switch/wp/2014/04/11/no-joke-this-company-wants-to-be-hacked-with-heartbleed/]

"Such is the haste to fix the fallout of the Heartbleed bug, some certificate authorities and website administrators have been making careless mistakes. PayPal's Hosted Message Applications, such as the one at https://view.paypal-communication.com, are now using Extended Validation certificates issued by VeriSign on 10 April 2014. The CAB Forum requires certificate authorities to adhere to a stringent set of guidelines [pdf] when issuing EV certificates, and it is the CA's responsibility to verify the accuracy of the information in the certificate. In particular, they must verify that the legal name of the subject in an EV certificate matches the name which appears on official government records. However, this verification does not appear to have been performed correctly in the case of these certificates."
[http://news.netcraft.com/archives/2014/04/11/heartbleed-certificate-revocation-tsunami-yet-to-arrive.html]

How can I protect myself?

Here is a list of all the password changing advice from the various companies and the results of a vulnerable scan performed on April 8, 12:00 UTC.

"Only 30,000 of the 500,000+ SSL certificates affected by the Heartbleed bug have been reissued up until today, and even fewer certificates have been revoked. Some of the first sites to deploy newly issued certificates in response to the OpenSSL vulnerability included Yahoo, Adobe, CloudFlare, DuckDuckGo, GitHub, Reddit , Launchpad, PayPal, Netflix and Amazon's CloudFront content delivery network."
[http://news.netcraft.com/archives/2014/04/11/heartbleed-certificate-revocation-tsunami-yet-to-arrive.html]

Fox-IT have actually listed a set of vulnerability tests, along with advice of the next steps and have developed a set of Snort signatures to detect successful exploitation of the bug.

Vendors are recommended to perform the following operations:

  • Upgrade the OpenSSL version to 1.0.1g (you can also recompile your OpenSSL 1.0.1 with the compile time option -DOPENSSL_NO_HEARTBEATS)
  • Request revocation of the current SSL certificate
  • Regenerate your private key
  • Request and replace the SSL certificate
  • Restart your services

Upgrading to OpenSSL 1.0.1g  does not automatically restart your services and likewise certificate revocation and replacement is a whole different step from ‘patching your server'.

  • If a vendor does not restart their services then their sessions are still vulnerable
  • Even if a vendor does revoke their certificates then some web browsers ignore the revocation
  • If a vendor does not replace their certificates then if their private key was stolen then they are wide open

If we look at the situation with WordPress then it appears they applied the patches but may not have replaced their certificates yet. I can also see smaller companies forgetting to restart their services after apply the OpenSSL 1.0.1g upgrade.

Should I change my password?

If you look at this list of vendors and their statements then the following rules apply:

  • You are safe if a vendor never used OpenSSL (remember SSL isn't broken itself, just one implementation of it including heartbeats)
  • You are safe if a vendor used an earlier version of OpenSSL (0.9.8 / 1.0.0)
  • You are moderately safe if a vendor upgraded to OpenSSL 1.0.1g, restarted their services and issued new certificates, before the public announcement on April 7th (apart from in the case of serious hackers)
  • You are at risk if a vendor upgraded to OpenSSL 1.0.1g, restarted their services and issued new certificates, sometime after April 7th (the longer the gap the higher the risk) and should change your passwords
  • You are at risk if a vendor upgraded to OpenSSL 1.0.1g, restarted their services and has not issued new certificates. You have a personal decision to make regarding changing passwords.

o   There is a small risk a private key could have been stolen and so even if you change your password then the site could still be vulnerable and you just gave your new password away

o   That said if the key did not get stolen then changing passwords prevents a hacker using your stolen details in the future.

  • You are at risk if a vendor either did not upgrade to OpenSSL 1.0.1g across all their services or forgot to restart their server. It is probably pretty pointless doing anything yet if that is the case.
  • You are at risk if a vendor made a nebulous statement like "We patched our servers" but was not drawn about restarting their services or issuing new certificates and should look to Internet forums
  • You are risk if a vendor has revoked their certificate but not issues a new certificate yet. Your web browser may ignore the certificate revocation and be open to MITM attacks.
  • You are at risk if you used client side software using a vulnerable version of OpenSSL and could have leaked confidential information from your desktop or mobile device

o   Double check the application that you use on Internet forums especially web browsers and IM clients

  • Most vendors are going to tell you to change your passwords anyway but check the details as above
  • Making your password really long is good practise but is no deterrent whatsoever from Heartbleed and you run the obvious risk of forgetting it or revealing it to hackers waiting for all the password resets

Imagine a hacker did target Vendor X, based on watching the charts of companies requesting new certificates and targeting those who haven't yet. Let us presume they were running Operating System X (all we know is that NGINX probably isn't vulnerable and Apache potentially is - but it would require specific timing) and just as a server rebooted the private key was added to the top of the OpenSSL memory stack and at that precise moment a hacker somehow managed to make a first request with an OpenSSL heartbeat and steal the private key. If we imagine that Vendor X are rebooting their servers regularly and maybe there was an insider involved then it becomes all the more probable. Let us now imagine the servers have been upgraded to OpenSSL 1.0.1g and all the services restarted but that the old certificates are still up there. That means if you reset your passwords then it makes no difference whatsoever because the hacker can decrypt all the traffic using their stolen private key. Changing your password actually means that the hacker now has your new password and you think you are actually safe. The hacker may have already decrypted everything on the site using their private key and have all your confidential information but let's assume they haven't. Vendor X then issue new certificates and does not bother going back through the last two years of logs to look for Heartbleed attacks. You have absolutely no idea that the hacker stole your new password and you carry on using the service and update it with your new credit card details...you are completely surprised that your account has been hacked even though Vendor X patched their servers.

That said, I would actually be more worried by the news that the majority of the traffic performing heartbeat vulnerability scans is apparently coming from various Government (thankfully not ours) sponsored cybercrime activity around the world.

I would like to end by thanking all the Blogs that I have referenced in bringing together this hopefully definitive guide to Heartbleed. I am happy to hear your views on the matter so please feel free to email me.

Peter

************************************************************************

Kevin Benedict Senior Analyst, Digital Transformation, EBA, Center for the Future of Work Cognizant View my profile on LinkedIn Learn about mobile strategies at MobileEnterpriseStrategies.com Follow me on Twitter @krbenedict Browse the Mobile Solution Directory Join the Linkedin Group Strategic Enterprise Mobility Join the Google+ Community Mobile Enterprise Strategies Recommended Strategy Book Code Halos Recommended iPad App Code Halos for iPads

***Full Disclosure: These are my personal opinions. No company is silly enough to claim them. I am a mobility and digital transformation analyst, consultant and writer. I work with and have worked with many of the companies mentioned in my articles.

More Stories By Kevin Benedict

Kevin Benedict is an opinionated Senior Analyst at Cognizant's Center for the Future of Work, SAP Mentor Alumnus, speaker, writer, and mobile and digital strategies expert. He is a popular keynote speaker, and in the past three years he has shared his insights into mobile and digital strategies with companies in 17 different countries. He has over 30 years of experience working with enterprise applications, and he is a veteran mobile industry executive. He wrote the Forward to SAP Press' bestselling book on enterprise mobility titled Mobilizing Your Enterprise with SAP, and he has written over 3,000 articles.

@ThingsExpo Stories
As businesses evolve, they need technology that is simple to help them succeed today and flexible enough to help them build for tomorrow. Chrome is fit for the workplace of the future — providing a secure, consistent user experience across a range of devices that can be used anywhere. In her session at 21st Cloud Expo, Vidya Nagarajan, a Senior Product Manager at Google, will take a look at various options as to how ChromeOS can be leveraged to interact with people on the devices, and formats th...
SYS-CON Events announced today that Yuasa System will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Yuasa System is introducing a multi-purpose endurance testing system for flexible displays, OLED devices, flexible substrates, flat cables, and films in smartphones, wearables, automobiles, and healthcare.
SYS-CON Events announced today that Dasher Technologies will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Dasher Technologies, Inc. ® is a premier IT solution provider that delivers expert technical resources along with trusted account executives to architect and deliver complete IT solutions and services to help our clients execute their goals, plans and objectives. Since 1999, we'v...
Enterprises have taken advantage of IoT to achieve important revenue and cost advantages. What is less apparent is how incumbent enterprises operating at scale have, following success with IoT, built analytic, operations management and software development capabilities – ranging from autonomous vehicles to manageable robotics installations. They have embraced these capabilities as if they were Silicon Valley startups. As a result, many firms employ new business models that place enormous impor...
SYS-CON Events announced today that Massive Networks, that helps your business operate seamlessly with fast, reliable, and secure internet and network solutions, has been named "Exhibitor" of SYS-CON's 21st International Cloud Expo ®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. As a premier telecommunications provider, Massive Networks is headquartered out of Louisville, Colorado. With years of experience under their belt, their team of...
SYS-CON Events announced today that Taica will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Taica manufacturers Alpha-GEL brand silicone components and materials, which maintain outstanding performance over a wide temperature range -40C to +200C. For more information, visit http://www.taica.co.jp/english/.
SYS-CON Events announced today that TidalScale, a leading provider of systems and services, will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. TidalScale has been involved in shaping the computing landscape. They've designed, developed and deployed some of the most important and successful systems and services in the history of the computing industry - internet, Ethernet, operating s...
SYS-CON Events announced today that MIRAI Inc. will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. MIRAI Inc. are IT consultants from the public sector whose mission is to solve social issues by technology and innovation and to create a meaningful future for people.
SYS-CON Events announced today that IBM has been named “Diamond Sponsor” of SYS-CON's 21st Cloud Expo, which will take place on October 31 through November 2nd 2017 at the Santa Clara Convention Center in Santa Clara, California.
SYS-CON Events announced today that TidalScale will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. TidalScale is the leading provider of Software-Defined Servers that bring flexibility to modern data centers by right-sizing servers on the fly to fit any data set or workload. TidalScale’s award-winning inverse hypervisor technology combines multiple commodity servers (including their ass...
Join IBM November 1 at 21st Cloud Expo at the Santa Clara Convention Center in Santa Clara, CA, and learn how IBM Watson can bring cognitive services and AI to intelligent, unmanned systems. Cognitive analysis impacts today’s systems with unparalleled ability that were previously available only to manned, back-end operations. Thanks to cloud processing, IBM Watson can bring cognitive services and AI to intelligent, unmanned systems. Imagine a robot vacuum that becomes your personal assistant tha...
Widespread fragmentation is stalling the growth of the IIoT and making it difficult for partners to work together. The number of software platforms, apps, hardware and connectivity standards is creating paralysis among businesses that are afraid of being locked into a solution. EdgeX Foundry is unifying the community around a common IoT edge framework and an ecosystem of interoperable components.
Infoblox delivers Actionable Network Intelligence to enterprise, government, and service provider customers around the world. They are the industry leader in DNS, DHCP, and IP address management, the category known as DDI. We empower thousands of organizations to control and secure their networks from the core-enabling them to increase efficiency and visibility, improve customer service, and meet compliance requirements.
As popularity of the smart home is growing and continues to go mainstream, technological factors play a greater role. The IoT protocol houses the interoperability battery consumption, security, and configuration of a smart home device, and it can be difficult for companies to choose the right kind for their product. For both DIY and professionally installed smart homes, developers need to consider each of these elements for their product to be successful in the market and current smart homes.
With major technology companies and startups seriously embracing Cloud strategies, now is the perfect time to attend 21st Cloud Expo October 31 - November 2, 2017, at the Santa Clara Convention Center, CA, and June 12-14, 2018, at the Javits Center in New York City, NY, and learn what is going on, contribute to the discussions, and ensure that your enterprise is on the right path to Digital Transformation.
Most technology leaders, contemporary and from the hardware era, are reshaping their businesses to do software. They hope to capture value from emerging technologies such as IoT, SDN, and AI. Ultimately, irrespective of the vertical, it is about deriving value from independent software applications participating in an ecosystem as one comprehensive solution. In his session at @ThingsExpo, Kausik Sridhar, founder and CTO of Pulzze Systems, will discuss how given the magnitude of today's applicati...
Smart cities have the potential to change our lives at so many levels for citizens: less pollution, reduced parking obstacles, better health, education and more energy savings. Real-time data streaming and the Internet of Things (IoT) possess the power to turn this vision into a reality. However, most organizations today are building their data infrastructure to focus solely on addressing immediate business needs vs. a platform capable of quickly adapting emerging technologies to address future ...
SYS-CON Events announced today that mruby Forum will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. mruby is the lightweight implementation of the Ruby language. We introduce mruby and the mruby IoT framework that enhances development productivity. For more information, visit http://forum.mruby.org/.
Digital transformation is changing the face of business. The IDC predicts that enterprises will commit to a massive new scale of digital transformation, to stake out leadership positions in the "digital transformation economy." Accordingly, attendees at the upcoming Cloud Expo | @ThingsExpo at the Santa Clara Convention Center in Santa Clara, CA, Oct 31-Nov 2, will find fresh new content in a new track called Enterprise Cloud & Digital Transformation.
SYS-CON Events announced today that NetApp has been named “Bronze Sponsor” of SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. NetApp is the data authority for hybrid cloud. NetApp provides a full range of hybrid cloud data services that simplify management of applications and data across cloud and on-premises environments to accelerate digital transformation. Together with their partners, NetApp emp...