Companies have an unparalleled opportunity to understand markets, consumers and competitors, innovate new products and reduce risk.
To exploit this opportunity requires sophisticated and comprehensive analysis of Big Data. Scalable high performance computing provides unrivalled breakthrough analysis provided in the most timely and cost-effective manner.
Data has been labelled the world’s most abundant resource. Making sense of the vast reservoirs of business, consumer, social and technical data that are becoming available holds the key to unlocking new opportunities in product development, competitive differentiation, financial risk management, sales and market positioning and process efficiencies.
Integrating and processing huge volumes of data is changing the dynamics of a range of business functions and operations including business intelligence and analytics, social and consumer behaviour analysis as well as applications in finance, engineering, life sciences and pharmaceuticals, manufacturing, and oil and gas exploration and production.
Exploiting all this data in a way that will put companies abreast or ahead of their competitors demands computing that is fully scalable and high performance engineered.
Scalability is the key
Scalable high performance computing (HPC) delivers an unrivalled depth of insights and breakthrough analysis on Big Data and data intensive workloads.
That’s a big claim: that scalable HPC cannot be rivalled for breakthrough analysis of Big Data and highly data-intensive workloads. Here’s another big claim: scalable HPC is far more cost-effective at Big Data analysis and data-intensive processing than any alternative platform.
To be effective as a business tool, any platform for Big Data analysis and data intensive computing has to be able to handle all the data available, carry out sophisticated, complex processing designed to discover hidden patterns and simulate complex environments and to do this rapidly and cost effectively.
It also has to be capable of being queried interactively to test different approaches or answer sequences of questions.
In a nutshell, any platform for Big Data has to big, fast, clever, cost-effective and capable of interactive querying. Which means that it has to be both scalable and high performance.
Only scalable high performance computing can deliver the full spectrum of fast, sophisticated analysis of very large data sets, making use of the most advanced algorithms for knowledge discovery on the largest, most complete and diverse data sets. And do it at speeds that can deliver near real time results, allowing managers and professionals to explore the conclusions and further interrogate the data. All for costs per analysis that are a fraction of those of alternative platforms.
Sophisticated business, financial, market and engineering algorithms typically have highly complex code, with extremely data dependent code paths and computation, with a great deal of indirection and with critical performance dependencies on memory latency. Scalable HPC platforms are engineered precisely to address these challenging demands which can bring other platforms to their knees.
There are commodity and cloud based platforms that claim scalability but without high performance compute capabilities.
There are also engineered high performance HPC platforms that do not feature scalability.
Commodity and cloud based platforms that attempt to scale over very large volumes of data can rapidly become very expensive to run. Processing times can stretch easily to hours, days or even weeks. Energy, storage and network costs can easily mount to become prohibitively expensive. TCO of such solutions is typically far higher than initially assumed.
Moreover, there is a heavy business cost of not delivering modelling, simulations or analysis in a timely fashion. The near real time capabilities of scalable HPC allow managers to formulate, question and model different approaches to the data, while engineering, product development and research teams can carry out multiple, complex simulations and design re-factors.
Such platforms are also frequently unable to perform sophisticated analysis or modelling of Big or complex data sets, and often require data sets to be restricted in size, complexity and heterogeneity. Examples where such platforms would struggle include market and consumer graph analytics, advanced business intelligence, financial analytics and integration, engineering prototyping and oil and gas reservoir analysis.
Given all these limitations, it is hard describe these platforms as in any sense truly scalable.
There are also high performance HPC platforms that are not scalable.
Without being optimised for data intensive computing and Big Data workloads, the performance of these platforms would be critically affected by memory, cache and storage I/O bottlenecks, and could also be affected by network restrictions as well. In such systems, the cost of moving data becomes very expensive, and the overall ROI of the system slumps as TCO soars.
What we are doing
In November 2015, the Linux Foundation launched the OpenHPC Collaborative project. This effort, which Lenovo is a founding member of, aims to provide a new, open source scalable framework to support the world’s most sophisticated high performance and scalable data intensive computing environments.
Lenovo is also a partner with Intel in its scalable system framework that will provide in 2016 an advanced converged architecture for HPC and Big Data which will enable a step change in high performance data intensive computing.
Lenovo is also partnering with HPC networking specialist to accelerate the adoption of EDR 100Gb/s Infiniband interconnect and server technologies to further optimise data intensive high performance computing.