About Searchable Storage

Searchable Storage is a managed storage solution that can be used within a Pipeline to manage, deduplicate, store, and search content. As Pipelines focus on processing the data live, in-pipeline storage can offer a number of benefits.

Performance and Capacity

Searchable Storage is designed to be instantly-responsive high-performance storage. Users of Searchable Storage have ingested entire social media Firehoses, and expanded storage to 5-10 terabytes without performance issues. Data in encrypted at rest and stored within the same Google Cloud environment as the Platform, ensuring safety and protection.

The Searchable Storage also handles many data performance and data maitenance tasks, such as: optimization of the data storage pattern, deduplication of content, and also augmenting stored data with updates if a newer version of duplicate content is received.

Use cases for Searchable Storage

The use cases for Searchable Storage are numerous, but here are some common use cases:

• Deduplication of content, especially those received from multiple sources.
• Direct usage as a search engine or key product database using available APIs
• Storing of content for batch or later processing.
• Buffer or backup to hold content in case of a customer's maintenance or outages in their own products.
• In cases of very complex requirements, as the egress of one pipeline, to then use it as the ingress of many others.
• Converting a high volume or highly variable firehose or real-time data source into a more manageable data stream.