appICU

by Woody Hutsell, http://www.appICU.com

A new analyst report on the use of flash as memory posted by David Floyer at Wikibon generated some attention in the market when it was covered by the Register. Floyer labels the architecture as FaME (Flash as Memory Extension). This blog discusses the merits of flash as memory architectures and how IBM approaches the flash as memory market.

The latency problems of disk have driven the IT industry toward a flock of solutions. One of course is flash storage, and the flash adoption rates show that this solution is rapidly gaining popularity. Another potential solution is to move more and more data into server memory (such as in-memory databases). But DRAM is volatile and relatively expensive. Flash is not volatile and much less expensive, but it’s of course slower than DRAM. In the middle of these two options lies the concept of “flash as memory.”

When we talk about using flash as memory we are wandering into interesting semantic territory. Flash is memory, so why do we need to have a new discussion about how to use flash as memory? In traditional use cases flash is used as storage, tucked away behind a block storage protocol, to allow flash memory to be easily integrated into traditional applications.

Using flash memory as memory, rather than as block storage, is gaining some traction in the marketplace. But using flash as memory requires some interesting trade-offs. Most importantly, we trade off latency (because RAM has much lower latency than flash) for much higher capacity/density and much lower cost per capacity. This latency trade-off runs counter to our typical reasons for using flash, which is to decrease the latency for data accesses vs hard disk drives. With flash as memory, we are increasing latency but for economic reasons.

For application developers, the choices have meaningful impacts. Relying on flash instead of disk means access to an infinitely large capacity of storage with low latencies. But these latencies are impacted by the processor to backplane latency + OS latency + file system latency + protocol latency + network latency + storage system latency. Various mitigating technologies are in play across each of these components, with different options affecting the total latency/cost/efficiency for the application. Relying on RAM, instead of disk, means that you get the lowest possible latency but with dramatic constraints on maximum capacity, acceptance of volatility, and the highest cost per capacity. Nonetheless, any “memory architecture” offers performance improvements on any models where hard disk drives are used as storage.

When application developers decide to adopt a flash-as-memory architecture, they can’t just plug in a flash system and expect the application to be plug and play. For application developers, using flash as memory means coding their applications to appropriate application program interfaces (API), which are likely to use memory access semantics instead of block access semantics. The development effort required to adopt a new API remains a significant limiting factor for broad marketplace adoption of these approaches in traditional applications, although efforts at standardization are emerging to lower that barrier. In most cases, application development with memory access semantics instead of block access semantics actually results in substantially simpler code. Once the application is coded to the API, then the experience for customers using the application vs an application previously running out of RAM is identical (recognizing some potential performance implications).

The question becomes: What is the best way to implement a flash-as-memory architecture? Is the best approach to use flash inside a server? Is the best approach to use PCI-attached flash? Is the best approach to use a flash appliance? Within each of these categories, solutions can be dramatically different regarding performance, reliability, and cost. Flash inside the server is fine for smaller capacity uses. If you decide to leave the boundaries of the server, the question becomes: What is the best way to connect an external flash appliance? There are a limited number of choices on the market today. Since 2014, IBM has offered the Data Engine for NoSQL – Power Systems Edition. There are many innovations in this particular flash-as-memory solution that have likely escaped the attention of the market. First, with the introduction of POWER8 technology, our Power Systems offerings now provide a new interface called CAPI (coherent accelerator processor interface) that cuts through many of the layers required in traditional x86 I/O designs. CAPI is an improvement on PCI Express used in traditional servers:

• CAPI allows the IBM FlashSystem 900 to interact with processors and system memory like another processor would
• This enables applications to access flash directly, bypassing the device driver, kernel, pinned pages, memory copies, etc.
• By removing the I/O subsystem overhead, the flash can be viewed as long-latency memory instead of as I/O-attached storage
• This eliminates >95% of the CPU cycles associated with moving data to and from flash, freeing up CPUs to do useful work (thus avoiding one of the pitfalls associated with other flash as memory solutions)
• The removal of code path length from the flash access reduces application-visible latency by more than a factor of 2 relative to accessing flash via the legacy I/O subsystem architecture
• The presence of a CAPI controller in the path to the flash enables future innovations which embed hardware-accelerated compute functionality in the flash read/write data path, leveraging the CPU efficiency and ease of programming that IBM’s CAPI architecture provides.

The second advantage of the Data Engine for NoSQL introduced above is that it uses IBM FlashSystem 900 as its flash memory repository. At this point, you are thinking – Aren’t all flash appliances created equal? What you should realize, of course, is that there are massive technology differences between flash appliances. IBM FlashSystem 900 is a product whose legacy was storing data in RAM. In RAM? Yes. For over 30 years Texas Memory Systems, the company IBM acquired to enter the flash memory business, sold systems based on RAM. Why does this matter? First, as I highlighted in my previous blog, our engineers are hard core when it comes to low latency. Our FlashSystem 900 is not polluted by latency-inducing storage services or bogged down by architectures originally designed for disk drives or even, for that matter, most flash drives – I don’t care whether that flash drive is attached with SAS, SATA, or NVMe. What IBM engineers do is inherently better because we started with an architecture that always treats flash as memory (remember we started with RAM) and then we just translate at the interface layer from whatever protocol we are attached to into direct memory accesses (DMA). A close look at the architecture reveals a flash appliance that does not use a PCI or NVMe backplane but an essentially native protocol-less backplane because we don’t want any software or artificial limits in the data path.

This architecture gives FlashSystem engineers endless flexibility, as demonstrated in our current array of solutions. This flexibility means that FlashSystem 900 can be used with the Data Engine for NoSQL in flash as memory use cases. It can also be used in traditional application acceleration environments where low latency storage is required, such as with Oracle RAC databases. It can be used in Tier 1 disk replacement architectures when used with IBM Spectrum Virtualize (SVC) or as part of our FlashSystem V9000. It can be used in scale-out object, block, and file use cases with our IBM Spectrum Scale solution. One elegantly defined system with multiple use cases.

The marketplace has not yet spoken on the importance of flash-as-memory. IBM with its Data Engine for NoSQL is a major early participant in this new storage direction, enabled by revolutionary foundational technologies such as CAPI and IBM FlashSystem – an end-to-end architecture only possible from a company whose reach spans from the server to the storage array.

by Woody Hutsell, http://www.appICU.com

It feels like I just participated in my billionth solid state storage product launch. In fact, I have participated in quite a few since the year 2000. In almost chronological order, here are the solid state storage systems I can recall launching into the market. If you are a solid state storage historian, you should pay close attention to the following table:

Some additional observations on trends over this period:

• Prices have come down from $5,000/GB in 2000 to below $10/GB in 2015. That means solid state storage costs 0.20% of what it cost 15 years ago.
• Capacity per rack unit in 2000 was 10 GB/RU. Capacity per rack unit in 2015 is 28,500 GB/RU. In the past 15 years solid state storage density has increased by 2,850x. Mind-blowing, really.
• Latency exhibited by the RAM-based products was very consistent and low over the 10 year period. Considerable cost, density, and persistence advantages forced a move to NAND flash memory in 2007, but latency for the flash-based products has remained consistently below 250 microseconds over the last 8 years.
• The continual trend to lower cost per bit NAND technologies (SLC to eMLC to MLC).
• There has been a consistent investment in enhanced data protection that started with protecting the memory in RAM using Chipkill and subsequently Variable Stripe RAID, a patented TMS technology.
• TMS, and now IBM, has improved RAS features as the systems evolved, from requiring two units for data protection to requiring only one highly available system.
• We’ve recently seen more demand for advanced storage services as the market for all flash arrays broadened from application acceleration to Tier 1 disk replacement.

Looking to make predictions about the future of solid state storage? A quick glance at the rate of product innovation and the reasons for change over the last 15 years can tell you much about how the industry will continue to evolve.

Ready or Not, Here Comes Flash.

Blog posted originally on:  http://datascopes.wordpress.com/ by Elan Freedberg

by Woody Hutsell, appICU

Wherever you are in your career, there is probably some place where you thought you would stay forever, a place where you developed the deepest roots or made the most impact. Things happen, you decide to move on, but you continue to feel a connection. For me, this was the case at Texas Memory Systems.

As those of you who have followed this blog know, I spent ten years of my life building the RAM and then flash systems business at Texas Memory Systems (TMS). Leaving in 2010 was a hard but necessary decision. When I left, I was not certain I would stay in the enterprise flash storage industry. In fact, a trip to the Flash Memory Summit in 2010 left me fairly certain the industry was not maturing in a way that was compelling, to me or to many enterprise storage buyers. Nonetheless, I still felt linked to and defined by my experience in the solid state storage business and started blogging through this site. I joined ViON and had a terrific experience selling enterprise storage to the enterprise, including solid state storage. I presented at the Flash Memory Summit in 2011 representing ViON and realized that I missed the industry more than I expected. More importantly, the industry was changing in some really interesting ways that made it enticing to return. I was presented with a great opportunity to work with Fusion-io on a software defined storage solution that we ended up calling the ION Data Accelerator.

In the background, the industry continued to mature, with the big companies finally moving into flash in a serious way. What started as OEM relationships and partnerships between companies was evolving into acquisitions. The most significant of these acquisitions was IBM’s decision to acquire Texas Memory Systems. Because my association with TMS reached back over 20 years, and because most of my friends in Houston were still employed at TMS, my interest in returning began to increase. As I listened to these friends and former co-workers, I realized that IBM was serious about this move into the flash market. I could see the culture and processes at TMS improving in ways I had literally dreamed that they could when I was at TMS years before. I gave the process a few months, just to make sure the post-acquisition honeymoon was not distorting opinions. Simultaneously, something really important started to happen; some of the really talented people that had left TMS were coming back and I could see that combined with the people IBM was bringing in, this new entity might really impact the market. The more people started returning to TMS, the more interested I became. The thing that finally clinched it was my realization that IBM was creating a flash division within the Systems and Technology Group less than two miles from my home. My decision was made and I am home again.

Here I am, nearly 15 months into the experience. I have to admit some initial fear about joining a big company like IBM. Will the TMS culture be crushed? Will the product suffer? Will IBM be too big and too slow to move fast in the flash market? Will the internal bureaucracy make the day-to-day work unpleasant? I definitely felt all of those concerns. Now, while there is no way to escape the size of IBM, I can say with conviction that most of my fears have not come true. But this was not without some struggle. The IBM team leading the integration has done everything in their power to preserve the culture and pace of TMS. IBM has done small things like preventing people from scheduling meetings over lunch hours so that the fairly notorious big lunch groups could continue. IBM has done big things like taking frequent surveys of employees to monitor their engagement, with real follow-through on issues identified. Yes, there is the back drop of a complex organization with some surging and some declining lines of business, but more than anything I see a depth of talent and resources that I consider to be unmatched.

The accolades that are accompanying IBM’s success in flash are rewarding, but it is really about the people and the journey. My compliments to the IBMers who moved to the FlashSystem team, and to the former TMSers who have worked harder than ever to deliver. I think sometimes the great people get lost in big organizations, so I want to recognize some spectacular contributions from Terri Mitchell, the IBM integration executive for fighting for this team; Jan Janick and Mike Nealon for preserving the engineering culture; Mike Kuhn for learning the flash market faster than I could have possibly imagined; Andy Walls for his architectural brilliance, as well as the hundreds of people on the FlashSystem team who are pushing every boundary and changing minds within and outside of IBM. This effort to become the unqualified leader in the all-flash array category is just beginning, but it is nice to get a great start with a strong team. Happy two year anniversary to the IBM Flash team!

This week I will be at IBM Enterprise 2014 Conference in Las Vegas presenting in three different sessions about FlashSystem. Also, make sure and visit http://www-03.ibm.com/systems/storage/flash/ for the latest product features coming October 6th.

Flash Industry Chaos

by Woody Hutsell, http://www.appICU.com

I’ve been watching the flash storage industry evolve since essentially its beginnings. A lot has changed in not so many years. Lately, though, I’ve seen some developments that suggest parts of the flash storage industry are in a period of rapid change, even chaos. And at the same time I’ve seen some clarity settle on the industry space closest to home.

In the last month, I visited two of the biggest shows for the enterprise flash market: Flash Memory Summit and VMworld. One of the things I like about going to these shows is keeping up with old friends, former co-workers, partners, analysts, and customers. I am used to seeing my colleagues in the industry move between companies (as I have done), but this year has really redefined the industry. The segment of the industry with the highest level of change is PCI flash. Last year set a prelude for changes in the server-based flash world with a few smaller deals like Virident selling to WD/HGST and Toshiba buying OCZ’s assets. This year, the two biggest players have changed hands, with both the LSI to Avago to Seagate and the Fusion-io to SanDisk deals closing.

From the perspective of its likelihood to disrupt the balance of power in the industry, the SanDisk acquisition of Fusion-io stands to make the biggest waves, or lead to the most chaos. Why do I think the SanDisk acquisition may lead to chaos? First, if SanDisk manages to retain the people that made Fusion-io special, SanDisk will gain a powerful new asset, but that isn’t easy. Buying a company for its strong engineering and sales talent means you must keep those people around. But, I have already seen a number of the excellent people at Fusion-io move on to other companies. Second, SanDisk now has the products and people necessary to sell directly to enterprise end users, even if the deals close through OEMs. They acquired a sales team that many competitors (think STEC, OCZ, Violin) tried to build but could not and were nearly bankrupted trying. But, this moves SanDisk perilously close to competing with its OEMs, which will create a fine line for them to walk, or fall from.

Another industry segment where chaos may be brewing is in VMware storage. At VMworld I saw a number of interesting new software defined storage solutions from VMware, plus a plethora of hyperconverged storage solutions like Nutanix. This part of the market has soaked up a lot of venture capital and it is apparent to me that the majority of the companies in this space will not make it. This year’s VMworld reminded me of previous Oracle World shows where you saw the small companies whose names you barely recognized buying the huge booth to try and capture some sliver of market attention. Almost inevitably, these companies crater rather than prosper. I think a really large booth when you are a small company with little revenue is the first evidence that you are on the path to irrelevance.

Finally, instead of rapid change and even chaos, I see one area within the flash storage industry that is gaining clarity – the all-flash array space. This industry category has reached its seventh birthday, if you consider that all-flash products were introduced to this space in 2007 by Texas Memory Systems. The recent round of Gartner Market Share and Magic Quadrant studies have confirmed what those in the industry realize – currently this is a three horse race, with IBM, EMC, and Pure Storage leading the industry in revenue and all three in the leadership quadrant. But it is clear to me that the other storage OEMs are gaining steam. Expect revenue from HDS, HP, and NetApp to increase on pace with industry growth. There continues to be a variety of small companies/start-ups that have missed out on the first wave of industry consolidation and are growing at a much smaller rate than the industry. For these companies, there is still a future if they can be acquired or grow into a profitable niche. It now takes much more as a startup (or established company) to enter and succeed in the AFA market than it took a year ago. The gap from the leaders to the followers in the AFA space continues to grow, and as that gap grows it becomes more important for clients to evaluate the long-term prospects of their flash array providers.

For more information on IBM FlashSystem, I encourage you to visit: http://www-03.ibm.com/systems/storage/flash/

by Woody Hutselll, http://www.appICU.com

The phrase “software defined storage” burst into the storage marketing lexicon in seemingly less time than the data access latency of a good SSD. Unless you were born yesterday, you saw it happen. Solid state storage vendors piled on the bandwagon, most of them leaping by the most convenient route. But IBM has taken a more reasoned, and seasoned, approach, resulting in a software defined storage solution that captures the benefits originally imagined in the phrase, without resorting to some quick-time-to-market strategies.

One of the more fascinating stories to me in the last two years has been the rapid adoption of the phrase: “software defined storage.” Here for your viewing pleasure is a Google Trends view:

The mainstream use of the term software defined storage started in August 2012 with the launch of the Fusion ION Data Accelerator. Within a few months every major and minor storage vendor was labeling their solution as software defined storage, including companies with solutions as different as Nexenta, NetApp, and IBM.

While researching for this blog, I came across a nice blog post by a fellow IBMer that casts additional light on the idea of software defined storage. I love that IDC created a software defined storage Taxonomy in April 2013. Can you believe it? From creation as a phrase, to requiring a taxonomy in less than eight months. If you are reading this, you can count yourself as having been along for the ride as this phrase started to infiltrate storage marketing.

As I explore the meaning of software defined storage, I will use a really basic definition that I think allows everyone to jump on the bandwagon:

Software-defined storage involves running storage services (such as replication, snapshots, tiering, virtualization and data reduction) on a server platform.

No wonder everyone can claim to be in the software defined storage business. Count IBM and its SAN Volume Controller (SVC) with over 10 years in the industry as a pioneer in this category. Certainly NetApp, Nexenta, and others belong as well. For years the storage industry has been migrating the delivery of storage services from custom-architected hardware to commodity server hardware. In doing so, vendors gain lower cost hardware, a faster time to market, and the advantage of using industry standard and open source software components. This isn’t to say the solutions aren’t differentiated; they are on the basis of their feature sets, but they are not significantly differentiated based on the hardware of their solution.

The introduction of all-Flash appliances into the product mix provided a real test of the capability of software defined storage. I remember IBM talking about project Quicksilver in 2008. Quicksilver used IBM SVC. The results were impressive and showed that software defined solutions could scale to IOPS levels required by the enterprise. Since that time nearly every Flash product brought to market could be labelled software defined storage: Intel server platform, Linux OS, software storage stack like SCST/LIO, HBAs/NICs, third party SSDs, and software for storage services. Storage has become integration and tuning rather than engineering. This approach to system design leaves a lot to be desired. Are the OS’s, storage stacks, RAID, enclosures, or HBAs all really designed for Flash? No, actually. The integration happens only in the minds of the marketers, unless you count the SAS link that connects the server to the storage enclosure or subsystem.

Instead, IBM has taken a novel approach to the Flash market, recognizing that producing extreme performance requires custom hardware, while also acknowledging that offering rich storage services is best accomplished with software defined storage. This recognition led IBM to offer a brand new solution called the FlashSystem V840 Enterprise Performance Solution. The software side of the equation is driven by IBM’s extensive experience building actual, integrated software defined storage solutions. The hardware side of the equation, rather than being a potpourri of third party stuff, is a custom-engineered Flash storage system (the IBM FlashSystem 840). On the software side, the software defined storage control modules have been purposely developed with data paths that substantially reduce the latency impact of most storage services. In fact, the FlashSystem V840 achieves latency for data accesses from Flash as low as 200 microseconds.

For a minute, let’s contrast the FlashSystem V840 with the attributes of nearly every competing Flash appliance offering:

Typical storage enclosure

• Third Party MLC/eMLC SSDs
•      No SSD-level data protection
•      Inexpensive processors as Flash controllers
•      SAS-connected
• Limited density and scalability due to form factor
• Off the shelf HBAs as interface controllers
• Software RAID and over-provisioning provided by the control enclosures

FlashSystem 840

• IBM designed FlashSystem Flash Modules
• IBM patented Variable Stripe RAID™ protects performance/availability even with cell, layer, or chip failures
• IBM engineered PowerPC processors combined with FPGAs as Flash controllers
• High speed proprietary interconnect
• High density and highly scalable
• IBM engineered interface controllers
• Optimized for low latency and high IOPS
• IBM engineered hardware RAID controllers
• Optimized for low latency and high IOPS with FPGAs as RAID controllers.

All of this discussion about proprietary hardware may have users worried about vendor lock-in and creating silos of data, however, the FlashSystem V840, with its storage virtualization feature, enables data center managers to break vendor lock-in by virtualizing heterogeneous third party arrays behind the FlashSytem V840 taking advantage of its feature rich set of storage services.

The choice of third party SSDs combined with software defined RAID architectures pushes storage processing work from the storage enclosure to the control enclosures. The problem is that these storage processing tasks are processor intensive (taking up threads and cores from what are already limited processors). The net result is that the control enclosures, without running any desirable storage services, are already burdened because they are performing functions that are best off-loaded to the storage enclosure. Combine this with the proven inefficiency of software RAID and the result is the terrible performance metrics we see from IBM’s Flash appliance competitors. Look closely at write IOPS performance and you will clearly see the deleterious effect of software RAID on performance. Try adding storage services to these control enclosures and you understand why the other Flash appliances on the market are not feature rich. Except by adding additional processors, they cannot add more features without cratering their already terrible performance.

In the case of the IBM FlashSystem V840, the storage enclosure functions as a high performance processing offload engine, freeing the control enclosures to do what they do best – implement storage services. The resulting solution delivers industry leading latency, IOPS, and bandwidth with a much more scalable solution.

Software defined storage may have its place, but only if done well. Abandoning effective hardware/software integration just for the chance to save on engineering seems like a terrible choice for all-Flash appliances. IBM has taken a different tack, purposely engineering and integrating a software defined storage solution that offers all the benefits, without resorting to the short-cuts that most storage vendors have used to get there.

To learn more about IBM and Software Defined Storage make sure and attend Edge2014.

Reaching the Summit
Woody Hutsell, AppICU

If you’ve worked for a small company you know making progress sometimes happens in baby steps. You deal with constrained resources. You deal with hasty or delayed decisions. You just deal with reality. We went through seven or so generations of RAM based SSD systems at TMS before we got to a solution I considered the pinnacle of achievement, the RamSan-440. It combined performance and reliability features that were the class of the industry. Even a year after flash solutions were released by TMS, I still recommended the RamSan-440 despite its higher cost per capacity.

You almost have to have been intimate to the all flash array business since its inception in 2007 to fully appreciate this next comment, but if you were and/or still are you will understand. Since the RamSan-500 and earliest competitor systems there has been a product engineering battle raging between competitors to develop the ultimate single box highly available all flash solution. Single box HA solutions are desirable for their improved density, performance, power and cost to deploy and support. This was a battle that could only be engaged by companies with real hardware engineering talent (a talent missing from most all-flash players today). For years, Tier 0 all flash arrays had to be deployed with 2x the capacity for 1x the usable capacity because of a range of issues including: lack of full redundancy in components, lack of hot swap components, lack of easy access to hot swap components and the inability of systems to have firmware upgrades without requiring downtime. These deficits resulted in a creative mix of deployment architectures to resolve the issue some more elegantly than others. Each product iteration has gotten us measurably closer to summiting this peak but not reached the peak. Similar issues surround competitor products with some farther behind than others. The achievement in the FlashSystem 840 is that it has reached the summit. I could not be happier with the product team who defined this product or the development team that brought it to market.

For more information on the FlashSystem 840, which IBM just announced yesterday, I encourage you to visit: http://www-03.ibm.com/systems/data/flash/storage/infographic/flash-data-center-optimized.html