Starfish Storage celebrates 10 years of unstructured data management

Starfish Storage, the sought-after entity in metadata-driven unstructured data management, marks its impressive 10-year journey. On this notable occasion, the company also revels in managing over an exabyte of capacity across its broad client base, including industries as diverse as pharmaceutical, automotive, media-entertainment, healthcare and many more. A considerable number of reputed institutions, such as seven of the eight Ivy League universities and Department of Energy supercomputing sites, are among Starfish's esteemed clients.

Starfish has effectively catered to the most intensive file environments around the globe. In general, a typical Starfish client deploys high-performance parallel file systems and scale-out NAS to service the requirements of high-demand workloads, such as scientific computing, AI/ML, engineering, rendering, etc. Starfish's clientele involves environments comprising billions of files, with capacity reaching up to hundreds of petabytes, successfully handling an array of data management challenges.

What sets Starfish apart from regular file management solutions is its metadata-driven modus operandi, which eschews dependence on mere timestamps and coarse analytics to derive policy conclusions. At the core of the Starfish platform lies an innovatively designed data catalogue specific for unstructured data.

The unique metadata and analytics potential of Starfish's catalogue facilitates servicing an extensive gamut of use cases. The platform empowers end users to manage their own data while maintaining the necessary security measures and safeguards, thereby addressing the intricate needs of every set of stakeholders involved.

Jacob Farmer, the Founder of Starfish Storage, highlights the complexity of managing large and diverse computing facilities where Starfish operates. He notes that within such environments, various use cases emerge, each presenting unique implementation challenges and intricacies that require careful consideration and attention to detail.

"In these large, diverse computing facilities where Starfish operates, multiple use cases appear with detailed implementation nuances," said Farmer. 

Farmer elaborates that a myriad of stakeholders are part of the process: "There are also many stakeholders including the users who create and consume the files as well those who are responsible for paying for storage capacity, various aspects of compliance, and ensuring proper data curation." 

Starfish Storage continues to support an abundant variety of use cases owing to its extremely adaptable architecture. This consists of a metadata-rich catalogue united with a scale-out data mover and batch processor. The platform's inner mechanics generate an automatic feedback loop between the catalogue and the batch processor, enabling automated data classification, metadata-driven storage management policies, user self-service and file-processing workflows, among others.

As it rounds up a prosperous decade, Starfish Storage anticipates another promising decade of growth and innovation in unstructured data management.