Reflections on Flash Memory Summit 2015

August 24, 2015

I just got back from my nth Flash Memory Summit. Special thanks to Tom Coughlin and the crew for putting on a good show and providing an excuse to get together with my friends in the industry.

I have some observations from the show this year:

1. There is nothing quite like a multi-billion dollar industry threatened by extinction to generate new breakthroughs, and 3D NAND technology looks like just the technology to extend the life of NAND flash for many more years. This means we can continue to project out density and cost improvements with NAND flash even as 3D NAND makes dealing with wear levelling a little bit easier for a generation or two.

2. We have been saying for years that “in five years” we would have a technology that could displace NAND flash. It looks like we will continue to be wrong, but the new announcement from Micron about 3D XPoint is nonetheless exciting because it may be our first viable storage class memory. There are a host of things people would like to do with NAND flash but they can’t because NAND is too slow to act as memory, or that they would like to do with RAM but RAM is volatile and low density. 3D XPoint appears to be a product that will enable some clever engineers to reach some markets poorly served by NAND flash.

3. Coincidental to the show, Pure Storage filed for a public offering. The financials that accompanies their filing made me pause and reflect that building a company to launch today is very different from building a company capable of long-term survival. At Texas Memory Systems (TMS), we did business the old fashioned way – we were profitable. The CEO never took on venture capital or long-term debt. His business could have continued indefinitely. The obvious downside of our approach at TMS was that we could not buy market attention and market share. To be relevant in our marketplace we had to produce the best technology. To be interesting to IBM, TMS had to have the best engineering. Now, it seems that the other start-ups in the industry that attempt to develop sustainable business models are mocked rather than celebrated. I think a few more market disappointments and with any luck we will learn to value businesses that build for a future. I believe the companies that are built to survive are better acquisition candidates.

4. The all-flash array market continues to be vibrant and fast growing. Even dropping $100 million from the 2014 market size estimates still shows a market in the early stages of spectacular growth. Just as interesting, we are starting to see companies aim for the edges of the market and position for promising new niches. The long awaited takeover of the data center by flash is well underway.

My final word of wisdom from this journey is that you should never take a cab from SFO to Santa Clara.

Flash as Memory

May 4, 2015

by Woody Hutsell,

A new analyst report on the use of flash as memory posted by David Floyer at Wikibon generated some attention in the market when it was covered by the Register. Floyer labels the architecture as FaME (Flash as Memory Extension). This blog discusses the merits of flash as memory architectures and how IBM approaches the flash as memory market.

The latency problems of disk have driven the IT industry toward a flock of solutions. One of course is flash storage, and the flash adoption rates show that this solution is rapidly gaining popularity. Another potential solution is to move more and more data into server memory (such as in-memory databases). But DRAM is volatile and relatively expensive. Flash is not volatile and much less expensive, but it’s of course slower than DRAM. In the middle of these two options lies the concept of “flash as memory.”

When we talk about using flash as memory we are wandering into interesting semantic territory. Flash is memory, so why do we need to have a new discussion about how to use flash as memory? In traditional use cases flash is used as storage, tucked away behind a block storage protocol, to allow flash memory to be easily integrated into traditional applications.

Using flash memory as memory, rather than as block storage, is gaining some traction in the marketplace. But using flash as memory requires some interesting trade-offs. Most importantly, we trade off latency (because RAM has much lower latency than flash) for much higher capacity/density and much lower cost per capacity. This latency trade-off runs counter to our typical reasons for using flash, which is to decrease the latency for data accesses vs hard disk drives. With flash as memory, we are increasing latency but for economic reasons.

For application developers, the choices have meaningful impacts. Relying on flash instead of disk means access to an infinitely large capacity of storage with low latencies. But these latencies are impacted by the processor to backplane latency + OS latency + file system latency + protocol latency + network latency + storage system latency. Various mitigating technologies are in play across each of these components, with different options affecting the total latency/cost/efficiency for the application. Relying on RAM, instead of disk, means that you get the lowest possible latency but with dramatic constraints on maximum capacity, acceptance of volatility, and the highest cost per capacity. Nonetheless, any “memory architecture” offers performance improvements on any models where hard disk drives are used as storage.

When application developers decide to adopt a flash-as-memory architecture, they can’t just plug in a flash system and expect the application to be plug and play. For application developers, using flash as memory means coding their applications to appropriate application program interfaces (API), which are likely to use memory access semantics instead of block access semantics. The development effort required to adopt a new API remains a significant limiting factor for broad marketplace adoption of these approaches in traditional applications, although efforts at standardization are emerging to lower that barrier. In most cases, application development with memory access semantics instead of block access semantics actually results in substantially simpler code. Once the application is coded to the API, then the experience for customers using the application vs an application previously running out of RAM is identical (recognizing some potential performance implications).

The question becomes: What is the best way to implement a flash-as-memory architecture? Is the best approach to use flash inside a server? Is the best approach to use PCI-attached flash? Is the best approach to use a flash appliance? Within each of these categories, solutions can be dramatically different regarding performance, reliability, and cost. Flash inside the server is fine for smaller capacity uses. If you decide to leave the boundaries of the server, the question becomes: What is the best way to connect an external flash appliance? There are a limited number of choices on the market today. Since 2014, IBM has offered the Data Engine for NoSQL – Power Systems Edition. There are many innovations in this particular flash-as-memory solution that have likely escaped the attention of the market. First, with the introduction of POWER8 technology, our Power Systems offerings now provide a new interface called CAPI (coherent accelerator processor interface) that cuts through many of the layers required in traditional x86 I/O designs. CAPI is an improvement on PCI Express used in traditional servers:

  • CAPI allows the IBM FlashSystem 900 to interact with processors and system memory like another processor would
  • This enables applications to access flash directly, bypassing the device driver, kernel, pinned pages, memory copies, etc.
  • By removing the I/O subsystem overhead, the flash can be viewed as long-latency memory instead of as I/O-attached storage
  • This eliminates >95% of the CPU cycles associated with moving data to and from flash, freeing up CPUs to do useful work (thus avoiding one of the pitfalls associated with other flash as memory solutions)
  • The removal of code path length from the flash access reduces application-visible latency by more than a factor of 2 relative to accessing flash via the legacy I/O subsystem architecture
  • The presence of a CAPI controller in the path to the flash enables future innovations which embed hardware-accelerated compute functionality in the flash read/write data path, leveraging the CPU efficiency and ease of programming that IBM’s CAPI architecture provides.

The second advantage of the Data Engine for NoSQL introduced above is that it uses IBM FlashSystem 900 as its flash memory repository. At this point, you are thinking – Aren’t all flash appliances created equal? What you should realize, of course, is that there are massive technology differences between flash appliances. IBM FlashSystem 900 is a product whose legacy was storing data in RAM. In RAM? Yes. For over 30 years Texas Memory Systems, the company IBM acquired to enter the flash memory business, sold systems based on RAM. Why does this matter? First, as I highlighted in my previous blog, our engineers are hard core when it comes to low latency. Our FlashSystem 900 is not polluted by latency-inducing storage services or bogged down by architectures originally designed for disk drives or even, for that matter, most flash drives – I don’t care whether that flash drive is attached with SAS, SATA, or NVMe. What IBM engineers do is inherently better because we started with an architecture that always treats flash as memory (remember we started with RAM) and then we just translate at the interface layer from whatever protocol we are attached to into direct memory accesses (DMA). A close look at the architecture reveals a flash appliance that does not use a PCI or NVMe backplane but an essentially native protocol-less backplane because we don’t want any software or artificial limits in the data path.

This architecture gives FlashSystem engineers endless flexibility, as demonstrated in our current array of solutions. This flexibility means that FlashSystem 900 can be used with the Data Engine for NoSQL in flash as memory use cases. It can also be used in traditional application acceleration environments where low latency storage is required, such as with Oracle RAC databases. It can be used in Tier 1 disk replacement architectures when used with IBM Spectrum Virtualize (SVC) or as part of our FlashSystem V9000. It can be used in scale-out object, block, and file use cases with our IBM Spectrum Scale solution. One elegantly defined system with multiple use cases.

The marketplace has not yet spoken on the importance of flash-as-memory. IBM with its Data Engine for NoSQL is a major early participant in this new storage direction, enabled by revolutionary foundational technologies such as CAPI and IBM FlashSystem – an end-to-end architecture only possible from a company whose reach spans from the server to the storage array.

Post Launch Reflections

April 7, 2015

by Woody Hutsell,

It feels like I just participated in my billionth solid state storage product launch. In fact, I have participated in quite a few since the year 2000. In almost chronological order, here are the solid state storage systems I can recall launching into the market. If you are a solid state storage historian, you should pay close attention to the following table:


Some additional observations on trends over this period:

  • Prices have come down from $5,000/GB in 2000 to below $10/GB in 2015. That means solid state storage costs 0.20% of what it cost 15 years ago.
  • Capacity per rack unit in 2000 was 10 GB/RU. Capacity per rack unit in 2015 is 28,500 GB/RU. In the past 15 years solid state storage density has increased by 2,850x. Mind-blowing, really.
  • Latency exhibited by the RAM-based products was very consistent and low over the 10 year period. Considerable cost, density, and persistence advantages forced a move to NAND flash memory in 2007, but latency for the flash-based products has remained consistently below 250 microseconds over the last 8 years.
  • The continual trend to lower cost per bit NAND technologies (SLC to eMLC to MLC).
  • There has been a consistent investment in enhanced data protection that started with protecting the memory in RAM using Chipkill and subsequently Variable Stripe RAID, a patented TMS technology.
  • TMS, and now IBM, has improved RAS features as the systems evolved, from requiring two units for data protection to requiring only one highly available system.
  • We’ve recently seen more demand for advanced storage services as the market for all flash arrays broadened from application acceleration to Tier 1 disk replacement.

Looking to make predictions about the future of solid state storage? A quick glance at the rate of product innovation and the reasons for change over the last 15 years can tell you much about how the industry will continue to evolve.

The Heart of FlashCore Technology

February 19, 2015

The Heart of FlashCore Technology

by Woody Hutsell,

So, it’s the 1980s or 1990s and you are building a solid state storage device with RAM. You can build it from scratch or you can take a computer with RAM, HDD and processors and a motherboard and turn it into a solid state storage device with a little bit of software. Which do you choose? If you’re in a hurry and have a short term view on the market, you take the fast way. How many companies built RAM solid state storage devices out of computers? Many, but zero that are still in business. Why are they out of business? Inferior product performance, inferior product cost structure, difficult to iterate without real engineering talent? All are reasonable answers. Many had their time in the spotlight but all disappear into history.

When the engineering team at Texas Memory Systems had to make this decision they did what any self-respecting engineering team would do, they designed a solid state storage device from scratch. No kidding. They never made the RAM chips, but they made just about everything else. Why would you do this? I wish you could have worked with some of the Texas Memory System customers in these days. Their interactions with the engineering team are nearly legendary around here. Could it be that customers would complain about a few microseconds when the rest of the world was dealing in 10s of milliseconds? There were and they did. And these engineers reacted and tuned and shaved off latency.

And that is where the story is kind of awesome, those engineers who dealt with those customers, those engineers are still at the heart, at the core, of the IBM engineering team for FlashSystem today. Now when someone comes in with a great idea, the first questions from these core engineers are what happens to the latency, what happens to the response time curve, how can we reduce the impact? This is not just an engineering goal, this is our engineering culture, the DNA that makes our products what they are.

So in marketing, we call it FlashCore Technology and we even subject it to another descriptor called Hardware Accelerated I/O but you know now that is really a lie. The core doesn’t start with Flash, it starts with people and a culture built meeting the demands of the most demanding customers in the world.

If you are looking for the technical depth in FlashCore look here.

Flash Riddle

January 7, 2015

by Woody Hutsell,

This isn’t Batman; this is your data center!

Riddle #1: What is a flash array that is fast like a Ferrari, has reliability and service like a Lexus, but is priced like a Chevrolet?

Answer: IBM FlashSystem

Riddle #2: What do you call the fastest, most feature rich but least expensive offering in a market?

Answer: The market leader in capacity shipments (see this link)

For as long as I have been associated with the solid state storage market, the products formerly known as Texas Memory Systems’ RamSan were labelled as the Ferrari of the market, but mostly in this context: “Who wouldn’t want to go that fast, but who can afford it?” For the most part, we embraced the label because we were the fastest. A quick look at Storage Performance Council results over the last decade can easily substantiate that position. But we did have a problem: The market didn’t perceive RamSan as the affordable choice, so we were left out of competitions before even being given a chance to compete. Who starts out their car buying process by verifying that the Ferrari is cost competitive? It was understood we were that fast and that expensive.

Since then, an interesting change has happened. IBM, with its buying power and economies of scale, has taken the Ferrari engine, surrounded it with Lexus-like reliability characteristics, and is now delivering it to the market with the lowest all-flash array price per capacity, according to some simple extrapolations from the latest IDC report on the state of the flash market.

Why is IBM throwing away its margins to take ownership of this market? It’s not. The economics are actually simple. IBM engineers the entirety of FlashSystem. As any accountant can tell you, this means that our R&D and engineering costs are going to be higher than the industry. But this is, in accounting terms, a fixed cost. If we pay this cost and don’t sell many products, we run at a loss. But if we pay this cost and sell a lot, our cost per unit only drops.

IBM buys NAND flash chips for FlashSystem; we don’t buy SSDs. Why does this matter? SSDs, in spite of their commodity nature and poor performance, are margin rich products for the companies that sell them. When our competitors buy SSDs to put in their all-flash arrays they are paying to someone else the margin needed to make investors happy while covering engineering investments. Thus, using SSDs actually makes the flash array product you buy more expensive. In accounting terms, SSDs represent a variable cost. As a vendor, you pay that same variable cost on every product you sell. Any business person will tell you it pays to decrease your variable costs because this enables you to bring your product to market for less cost than your competitors. This is especially important when you’re selling at the kinds of volumes where IBM sells in the all-flash array market – more than the next two competitors combined in the first half of this year, according to the same IDC report noted above. This explains why we are indeed a leader in this market space.

Maybe not what you’d expect from a company with an enterprise-grade reputation like IBM.

So, what does this mean to our clients and potential clients? FlashSystem can save you money. But the advantages don’t stop there.

Did you know FlashSystem offers inline compression and what’s more, testing of our inline compression at customer sites shows that it can be more effective and faster than that of our competitors? As a potential customer, there is a simple way for you to find out if this is true for your workload – include FlashSystem in your next storage procurement evaluation.

You could pay more and get less, but why should you? That’s a riddle worth answering.

Ready or Not, Here Comes Flash

November 17, 2014

Ready or Not, Here Comes Flash.

Blog posted originally on: by Elan Freedberg

Coming Home

October 5, 2014

by Woody Hutsell, appICU

Wherever you are in your career, there is probably some place where you thought you would stay forever, a place where you developed the deepest roots or made the most impact. Things happen, you decide to move on, but you continue to feel a connection. For me, this was the case at Texas Memory Systems.

As those of you who have followed this blog know, I spent ten years of my life building the RAM and then flash systems business at Texas Memory Systems (TMS). Leaving in 2010 was a hard but necessary decision. When I left, I was not certain I would stay in the enterprise flash storage industry. In fact, a trip to the Flash Memory Summit in 2010 left me fairly certain the industry was not maturing in a way that was compelling, to me or to many enterprise storage buyers. Nonetheless, I still felt linked to and defined by my experience in the solid state storage business and started blogging through this site. I joined ViON and had a terrific experience selling enterprise storage to the enterprise, including solid state storage. I presented at the Flash Memory Summit in 2011 representing ViON and realized that I missed the industry more than I expected. More importantly, the industry was changing in some really interesting ways that made it enticing to return. I was presented with a great opportunity to work with Fusion-io on a software defined storage solution that we ended up calling the ION Data Accelerator.

In the background, the industry continued to mature, with the big companies finally moving into flash in a serious way. What started as OEM relationships and partnerships between companies was evolving into acquisitions. The most significant of these acquisitions was IBM’s decision to acquire Texas Memory Systems. Because my association with TMS reached back over 20 years, and because most of my friends in Houston were still employed at TMS, my interest in returning began to increase. As I listened to these friends and former co-workers, I realized that IBM was serious about this move into the flash market. I could see the culture and processes at TMS improving in ways I had literally dreamed that they could when I was at TMS years before. I gave the process a few months, just to make sure the post-acquisition honeymoon was not distorting opinions. Simultaneously, something really important started to happen; some of the really talented people that had left TMS were coming back and I could see that combined with the people IBM was bringing in, this new entity might really impact the market. The more people started returning to TMS, the more interested I became. The thing that finally clinched it was my realization that IBM was creating a flash division within the Systems and Technology Group less than two miles from my home. My decision was made and I am home again.

Here I am, nearly 15 months into the experience. I have to admit some initial fear about joining a big company like IBM. Will the TMS culture be crushed? Will the product suffer? Will IBM be too big and too slow to move fast in the flash market? Will the internal bureaucracy make the day-to-day work unpleasant? I definitely felt all of those concerns. Now, while there is no way to escape the size of IBM, I can say with conviction that most of my fears have not come true. But this was not without some struggle. The IBM team leading the integration has done everything in their power to preserve the culture and pace of TMS. IBM has done small things like preventing people from scheduling meetings over lunch hours so that the fairly notorious big lunch groups could continue. IBM has done big things like taking frequent surveys of employees to monitor their engagement, with real follow-through on issues identified. Yes, there is the back drop of a complex organization with some surging and some declining lines of business, but more than anything I see a depth of talent and resources that I consider to be unmatched.

The accolades that are accompanying IBM’s success in flash are rewarding, but it is really about the people and the journey. My compliments to the IBMers who moved to the FlashSystem team, and to the former TMSers who have worked harder than ever to deliver. I think sometimes the great people get lost in big organizations, so I want to recognize some spectacular contributions from Terri Mitchell, the IBM integration executive for fighting for this team; Jan Janick and Mike Nealon for preserving the engineering culture; Mike Kuhn for learning the flash market faster than I could have possibly imagined; Andy Walls for his architectural brilliance, as well as the hundreds of people on the FlashSystem team who are pushing every boundary and changing minds within and outside of IBM. This effort to become the unqualified leader in the all-flash array category is just beginning, but it is nice to get a great start with a strong team. Happy two year anniversary to the IBM Flash team!

This week I will be at IBM Enterprise 2014 Conference in Las Vegas presenting in three different sessions about FlashSystem. Also, make sure and visit for the latest product features coming October 6th.