How big data is changing the oil & gas industry

Article Index

Everyone needs it, few know how we get it, and many feel compelled to slow down efforts to finding and producing oil. One of the primary assets of successful, thriving societies is a low-cost energy source. What drives low cost? Supply greater than demand! What drives supply? Finding supplies in sufficient quantities so producing oil and gas is economically viable.

Finding and producing hydrocarbons is technically challenging and economically risky. The process generates a large amount of data, and the industry needs new technologies and approaches to integrate and interpret this data to drive faster and more accurate decisions. Doing so will lead to safely finding new resources, increasing recovery rates and reducing environmental impacts.

The term “big data” has historically been regarded by the oil and gas industry as a term used by “softer” industries to track people’s behaviors, buying tendencies, sentiments, etc. However, the concept of “big data” – defined as increasing volume, variety and velocity of data – is quite familiar to the oil and gas industry.

The processes and decisions related to oil and natural gas exploration, development and production generate large amounts of data. The data volume grows daily. With new data acquisition, processing and storage solutions – and the development of new devices to track a wider array of reservoir, machinery and personnel performance – today’s total data is predicted to double in the next two years.

Many types of captured data are used to create models and images of the Earth’s structure and layers 5,000-35,000 feet below the surface and to describe activities around the wells themselves, such as machinery performance, oil flow rates and pressures. With approximately one million wells currently producing oil and/or gas in the United States alone, and many more gauges monitoring performance, this dataset is growing daily.

The oil industry recognizes that great power and imminent breakthroughs can be found in this data by using it in smarter, faster ways. However, resistance regarding workflows and analysis approaches remains in place, as it has for the last 30 years. How does the industry bridge the vocabulary and cultural gap between data scientists and technical petroleum professionals? Ideas, applications and solutions generated outside the oil and gas industry rarely find their way inside. Other industries seem to have bridged this gap, but in talking to experts in the broader technology industry, the oil industry is seen as a “no man’s land” for new-age entrepreneurs, while major technology providers spend billions trying to enter it (e.g., GE, IBM and Microsoft).

Breaking into the oil and gas industry is difficult for analysts, but the need and potential for reward are great. Nine of the top 10 organizations in Fortune’s Global 500 are oil and gas companies. More than 20,000 companies are associated with the oil business, and almost all of them need data analytics and integrated technology throughout the oil and gas lifecycle.

Throughout the 1990s, the oil and gas industry focused on data integration, i.e., How do we get all the data in one place and make it available to the geo-scientists and engineers working to find and produce hydrocarbons? Since the turn of the century, technology development has mainly focused on software that integrates across the major disciplines to speed up old workflows. The industry has had many amazing technical professionals, but the idea of a “data scientist” is new, and should be considered alongside the petrophysical, geophysical and engineering scientists. The next decade must focus on ways to use of all of the data the industry generates to automate simple decisions and guide harder ones, ultimately reducing the risk and resulting in finding and producing more oil and gas with less environmental impact.


Figure 1: Simple seismic acquisition diagram (left) and a processed, interpreted 3D seismic Earth model (right).


Technically Complex, High Risk

Despite its astronomical revenues, the profit margin of the oil and gas majors is 8 percent to 9 percent. Finding and developing oil and gas while reducing the safety risk and environmental impact is difficult. The layers of hydrocarbon-bearing rock are deep below the Earth’s surface, with much of the world’s hydrocarbons locked in hard-to-reach places, such as in deep water or areas with difficult geopolitics.


Oil is not found in big, cavernous pools in the ground. It resides in layers of rock, stored in the tiny pores between the grains of rock. Much of the rock containing oil is tighter than the surface on which your computer currently sits. Further, oil is found in areas that have structurally “trapped” the oil and gas – there is no way out. Without a structural trap, oil and gas commonly “migrates” throughout the rock, resulting in lower pressures and uneconomic deposits. All of the geological components play an important role; in drilling wells, all components are technically challenging.


Following are three big oil industry problems that consume money and produce data:

1. Oil is hard to find. Reservoirs are generally 5,000 to 35,000 feet below the Earth’s surface. Low-resolution imaging and expensive well logs (after the wells are drilled) are the only options for finding and describing the reservoirs. Rock is complex for fluids to move through to the wellbore, and the fluids themselves are complex and have many different physical properties.

2. Oil is expensive to produce. The large amount science, machinery and manpower required to produce a barrel of oil must be done profitably, taking into account cost, quantity and market availability.

3. Drilling for oil presents potential environmental and human safety concerns that must be addressed.

Finding and producing oil involves many specialized scientific domains (i.e., geophysics, geology and engineering), each solving important parts of the equation. When combined, these components describe a localized system containing hydrocarbons. Each localized system (reservoir) has a unique recipe for getting the most out of the ground profitably and safely.