I have multiple front end workstation level computers to access the data that I need to work with and a major size video wall to display all the data on.

In general there are hundreds of servers running Open Solaris set up into a four tier data stage. Each level has maximum size caches to minimize their own disk drive access.

Utilizing the snapshot capability in ZFS it is very easy to migrate data on a LRU basis from one tier to the next.

The most current data is in the front end tier. When the drives in the pool become 90% full, stuff is moved up to the next tier, verified as transferred and then deleted after verification until the pool is 70% or less full.

Each tier is larger than the one under it. Therefore by the theory of working sets the chance of a hit grows exponentially as you move from one tier to the next.

Two operations go on her in parallel. The active usage servers take in the feeds and run a massive set of distributed models to look at the data and present what I want for my purposes.

Then the last tier of data is used with heavy duty mainframe (IBM full loaded Z9 series computers) to back test the models and recommend feedback for adjusting the tuning and weighting parameters in the models to heuristically tune in on the optimal solution for the models I run.

When you get into data running in my server farm hitting thousands of TB and getting ready to push into the PB range the flexibility of ZFS and disk pools without having to resort to all sorts of volume managers and such is awesome.

I also really like the fact that ZFS gets rid of a lot of holes in hardware raid controllers and silent disk errors.

As it uses copy on write to deal with transactions and the pool size can be as big as you want , you end up with faster access as the numbers of drives grow, since most of your 'working set' of data is already under the heads and doesn't require seeks.

With good SSD caches or front end ram based cache systems you can literally end up with only a few percent of your queries even going out to the disk farm.