Assess
Directional Fit Score
Fifteen weighted factors produce a 0–100 score. Every factor shows its weight, rating and reasoning, so the result can be explained — and challenged — in front of a customer. This is a qualification aid, not a benchmark.
Comma separated. Recommendations preserve these systems rather than replacing them.
What changed in the last few months. Drives urgency, discovery and architecture.
85/100
Strong directional fit — pursue with a scoped technical validation.
A qualification aid, not a benchmark, prediction or guarantee. Thresholds used here are heuristics, not published product limits.
How the score was calculated
This workload is a canonical real-time analytics pattern with multiple proven references.
500 GB/day is comfortably above the threshold where compression and pre-aggregation pay for themselves.
Sub-second interactive latency is the clearest differentiator against warehouses.
Observability spend growth is a well-worn entry point.
Second-level freshness is a core design point, not an add-on.
2 pain drivers identified: Cost growth, Retention limits.
150 concurrent readers is where warehouse queueing and credit burn typically appear.
$90,000/month is a solid base for a business case.
High cardinality is where index-based and per-metric-billed tools break down or become punitive.
60% YoY growth creates urgency even where current pain is tolerable.
SaaS has established real-time analytics precedent and peer references.
12 months of retention creates a meaningful storage cost lever.
Important-but-not-critical is the easiest risk profile for a first workload.
Internal-only usage means the business case rests on cost and productivity rather than revenue.
Moderate engineering burden supports but does not carry the case.
Why now?
Observability / SIEM bill increased sharply. A dated renewal or budget line creates a decision deadline that does not depend on us.
- Observability / SIEM bill increased sharply — Frame the business case as cost per retained GB and per retained month, not as a platform swap.
- Check whether any roadmap item exposes this data to customers within a year.
Architecture workbench
Current state, target state, what stays, what changes, migration sequence, risks and POC criteria — derived from the same inputs.
Target state
- 01Services & infrastructure
- 02OpenTelemetry collector
- 03ClickHouse (ClickStack)
- 04HyperDX / Grafana / SQL
- 05Alerting & on-call
Current state
- ·Today's stack: Datadog, Kafka, Snowflake.
- ·Workload: observability at roughly 500 GB/day, 12 months retained.
- ·Freshness today is treated as seconds; query expectation is sub second.
- ·Telemetry is collected by agents into a per-signal commercial platform
- ·Retention is trimmed to control cost rather than to match need
- ·Correlating logs, metrics and traces means moving between tools
What stays
- ·On-call workflow and incident management if already embedded
- ·APM for a small set of critical services during transition
- ·Synthetic monitoring and RUM if not being replaced
- ·Datadog keeps its current role unless a specific reason to change it is agreed.
- ·Kafka keeps its current role unless a specific reason to change it is agreed.
- ·Snowflake keeps its current role unless a specific reason to change it is agreed.
What changes
- ·Collection standardises on OpenTelemetry rather than vendor agents
- ·Signals land in one queryable store with a single retention policy
- ·Retention becomes a cost decision rather than a platform constraint
Why this pattern
- ·Vendor-neutral collection removes proprietary agent lock-in.
- ·Logs, metrics and traces become joinable in one SQL store.
- ·Retention economics change from days to months.
- ·Cardinality is a storage question rather than a billing event.
- ·Observability / SIEM bill increased sharply: Frame the business case as cost per retained GB and per retained month, not as a platform swap.
Executive summary
Telemetry collection moves to an open standard and a single store, so retention and cost become choices the business controls rather than constraints set by tooling.
Technical summary
OpenTelemetry collectors ship logs, metrics and traces into ClickHouse-backed observability, with one storage and retention model across signals.
Target state
- 01Producers
- 02Kafka / Confluent / Kinesis
- 03ClickHouse ingest (Kafka engine or ClickPipes)
- 04Materialized views → rollups
- 05Real-time apps & dashboards
Current state
- ·Today's stack: Datadog, Kafka, Snowflake.
- ·Workload: observability at roughly 500 GB/day, 12 months retained.
- ·Freshness today is treated as seconds; query expectation is sub second.
- ·Events already land in Kafka but are consumed into a slower store
- ·Stream processing jobs pre-aggregate because the sink cannot keep up
- ·Late or replayed events are difficult to correct
What stays
- ·Kafka remains the transport and buffer
- ·Stream processing jobs that enrich or join before landing
- ·Other consumers of the same topics
- ·Datadog keeps its current role unless a specific reason to change it is agreed.
- ·Kafka keeps its current role unless a specific reason to change it is agreed.
- ·Snowflake keeps its current role unless a specific reason to change it is agreed.
What changes
- ·ClickHouse consumes directly from the existing topics
- ·Raw events are retained alongside aggregates rather than discarded
- ·Some stream-processing logic collapses into materialised views
Why this pattern
- ·Reuses existing pipelines — no new data collection layer.
- ·Batched consumption keeps insert efficiency high.
- ·Materialized views pre-compute the expensive work once per row.
- ·Raw events remain queryable alongside aggregates for investigation.
- ·Observability / SIEM bill increased sharply: Frame the business case as cost per retained GB and per retained month, not as a platform swap.
Executive summary
Streams already flowing through the business become directly queryable, so teams can ask questions of live data instead of waiting for a downstream copy.
Technical summary
ClickHouse consumes Kafka topics natively, retaining raw events with materialised views maintaining incremental aggregates for serving.
Mintlify · Developer documentation SaaS
A small team shipping customer-facing analytics without a data platform org.
- · Customer-facing analytics shipped by a small team
- · Predictable cost as customers scaled
Cloudflare · Internet infrastructure
One of the largest public examples of ClickHouse running customer-facing analytics at internet scale.
- · Internet-scale event ingest with interactive query latency
- · Customer-facing analytics served from the same platform as internal analysis
Lyft · Mobility / marketplaces
Large-scale marketplace operations backed by real-time analytics rather than batch reporting.
- · Interactive queries over high-volume operational data
- · Reduced cost per GB of telemetry retained
Huntress · Managed security (MDR)
A security vendor storing and hunting over full telemetry rather than sampled logs.
- · More telemetry retained for longer
- · Faster analyst investigation across the estate
Walk me through the workload that made you look at this — what data, what users, what question are they asking?
Workload shape determines fit far more than industry or company size.
What does a user experience today — how long between clicking and seeing results?
Establishes the measurable baseline any business case will be judged against.
How much data lands per day now, and what does your growth plan imply for eighteen months out?
Trajectory usually creates the urgency that today's numbers do not.
What are you sampling, dropping or shortening retention on today purely to control cost?
Reveals hidden cost pain even when the invoice is described as acceptable.
Which line item in the data platform bill is growing fastest, and who owns that budget?
Finds the economic buyer and the number the business case must move.
How much engineering time per month goes into working around the current platform's limits?
Quantifies the soft cost that usually dwarfs the licence line.
What's between the source systems and the analytics layer today — streams, CDC, batch jobs?
Determines the architecture pattern and realistic time-to-value.
If we did a pilot, what specific numbers would make it a success in your leadership's eyes?
Turns interest into measurable, defensible exit criteria.
Who else needs to agree, and what has your organisation's last infrastructure decision looked like?
Surfaces the real decision process and procurement path early.
How many distinct entities — customers, devices, symbols, tenants — do you need to slice by?
High cardinality is where incumbent tools usually break down or bill punitively.