Storage Vendor News
Hu Yoshidas blog
DataOps and Hitachi Vantara
Thu, 11 Apr 2019
According to the Harvard Business Review, "Cross-industry studies show that on average, less than half of an organization’s structured data is actively used in making decisions—and less than 1% of its unstructured data is analyzed or used at all. More than 70% of employees have access to data they should not, and 80% of analysts’ time is spent simply discovering and preparing data. Data breaches are common, rogue data sets propagate in silos, and companies’ data technology often isn’t up to the demands put on it." That was in a report back in 2017. What has changed since then?
Few Data Management Frameworks are Business Focused
Data management has been around since the beginning of IT, and a lot of technology has been focused on big data deployments, governance, best practices, tools, etc. However, large data hubs over the last 25 years (e.g., data warehouses, master data management, data lakes, Hadoop, Salesforce and ERP) have resulted in more data silos that are not easily understood, related, or shared. Few if any data management frameworks are business focused, to not only promote efficient use of data and allocation of resources, but also to curate the data to understand the meaning of the data as well as the technologies that are applied to the data so that data engineers can move and transform the essential data that data consumers need.
Today more customer are focusing on the operational aspects of data rather than on the fundamentals of capturing, storing and protecting data. Following the success of DevOps (a set of practices that automates the processes between software development and IT teams, in order that they can build, test, and release software faster and more reliably) companies are now focusing on DataOps. DataOps can best be described by Andy Palmer, who coined the term in 2015, “The framework of tools and culture that allow data engineering organizations to deliver rapid, comprehensive and curated data to their users … [it] is the intersection of data engineering, data integration, data quality and data security. Fundamentally, DataOps is an umbrella term that attempts to unify all the roles and responsibilities in the data engineering domain by applying collaborative techniques to a team. Its mission is to deliver data by aligning the burden of testing together with various integration and deployment tasks.”
At Hitachi Vantara we have been applying our technologies to DataOps in four areas: Hitachi Content Platform, Pentaho, Enterprise IT Infrastructure, and REAN Cloud.
- HCP: Object storage for unstructured data through our Hitachi Content Platform and Hitachi Content Intelligence software. Object storage with rich meta data, content intelligence, data integration, and analytics orchestration tools enable business executives to identify data sources, data quality issues, types of analysis and new work practices needed to use those insights
- Pentaho: Pentaho streamlines the entire machine learning workflow and enables teams of data scientists, engineers and analysts to train, tune, test and deploy predictive models.
- IT Infrastructure: Secure Enterprise IT Infrastructure that extends across edge to core to Cloud, based on REST APIs for easy integration with third party vendors. This gives us the opportunity to not only connect with other vendor’s management stacks like ServiceNow, but also apply analytics and machine learning and automate deployment of resources through REST APIs.
- REAN Cloud: A cloud agnostic managed services platform for DataOps in the cloud. Highly differentiated offerings to migrate applications to the cloud, modernize applications to leverage the cloud offerings for data warehouse modernization, predictive agile analytics, and real time IoT. REAN Cloud also provides ongoing managed services.
- Big Data systems are becoming a center of gravity in terms of storage, access and operations.
- Businesses are looking to DataOps, to speed up the process of turning data into business out comes.
- DataOps is needed to understand the meaning of the data as well as the technologies that are applied to the data so that data engineers can move, automate and transform the essential data that data consumers need.
- Hitachi Vantara provides DataOps tools and platforms through
- Hitachi Content Platform,
- Pentaho data integration and analytics orchestration,
- Infrastructure analytics and automation
- REAN Cloud migration, modernization, and managed services.
Grad Student Katie Bouman uses DataOps to capture first picture of a black hole.
Tastes Like Chicken
Tue, 02 Apr 2019
In 2050, the population of the world is expected to be 9 billion versus the 7 billion today. The challenge will be to feed 2 billion more people with less arable land, less water, and less farmers. One of the increasing demands will be for protein as people demand richer foods.
If you are over 50 years old, you may remember a satirical American comic strip that appeared in many newspapers in the United States, Canada and Europe, featuring a fictional clan of hillbillies in the impoverished mountain village of Dogpatch, USA. Written and drawn by Al Capp. In one of the episodes of this series, the young hero Lil Abner discovers the Shmoo in a hidden valley and introduces them to Dog Patch and the rest of the world. The Shmoo was a lovable creature that laid eggs, gave milk, loved to be eaten and tasted like any meat desired, chicken when fried, steak when broiled, pork when roasted, and catfish when baked. They multiplied like rabbits but required no feed or water, only air to breath. The perfect solution to world hunger.
Today we have something that is close to the Shmoo. That is today’s broiler chicken. In 1957 the average chicken weighed about 1 KG or 2.2 lb. Today a commercially grown broiler chicken weighs 9.3 lbs. after 8 weeks. It only takes 2.5 lbs. of feed and 468 gallons of water to produce one lb. of chicken meat, which is much more efficient than the production of a lb. of pork or beef, with much less waste, less space and less CO2 emissions.
IT appears that chicken will be the meat for the masses.
IoT will help to increase agriculture efficiencies, reduce spoilage, and increase the freshness and nutritional content of healthy foods. The problem will not be about the production of foods, but how to build the infrastructure to provide equal access to that food to all 9 billion people. According to a BoA, Merrill Lynch, Global Investment Strategy report, populous countries like Nigeria, Pakistan and Kenya spend 47 to 57% of their household expenditure on food compared to 7% in the US and the UK.
To finish the story of the Shmoo. The Shmoo became so popular that people no longer needed to go to the stores to buy food. This caused a series of images reminiscent of the Wall Street Crash of 1929, and the Captains of Industry banded together to exterminate the Shmoo. Two of the Shmoo managed to escape to go back to their hidden valley in the mountains. Wikipedia described the Shmoo sequence as “massively popular, both as a commentary on the state of society and a classic allegory of greed and corruption tarnishing all that is good and innocent in the world. In their very few subsequent appearances in Li'l Abner, Shmoos are also identified by the U.S. military as a major economic threat to national security.”
Mr. Higashihara, our Hitachi CEO, always reminds us of the Light and Shadow of Digital Transformation. With every advancement in digital transformation we must be mindful of the possible shadows which may negate our vision for social innovation.