Whom the Gods Would Destroy, They First Give Real-time Analytics

Mike's Notes

Another take on web analytics by Dan McKinley, Principal Engineer at Etsy.

Resources

• https://mcfunley.com/whom-the-gods-would-destroy-they-first-give-real-time-analytics
• http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/
• http://www.evanmiller.org/how-not-to-run-an-ab-test.html
• https://en.wikipedia.org/wiki/CAP_theorem
• https://www.infoq.com/articles/cap-twelve-years-later-how-the-rules-have-changed/

References

• 
  

Repository

• Home > Ajabbi Research > Library

Last Updated

17/03/2025

Whom the Gods Would Destroy, They First Give Real-time Analytics

By: Dan McKinley

mcfunley.com: January 9th, 2013

Homer: There's three ways to do things. The right way, the wrong way, and the Max Power way!

Bart: Isn't that the wrong way?

Homer: Yeah. But faster!

- "Homer to the Max"

Every few months, I try to talk someone down from building a real-time product analytics system. When I’m lucky, I can get to them early.

The turnaround time for most of the web analysis done at Etsy is at least 24 hours. This a ranking source of grousing. Decreasing this interval is periodically raised as a priority, either by engineers itching for a challenge or by others hoping to make decisions more rapidly. There are companies out there selling instant usage numbers, so why can’t we have them?

Here’s an excerpt from a manifesto demanding the construction of such a system. This was written several years ago by an otherwise brilliant individual, whom I respect. I have made a few omissions for brevity.

We believe in…

1. Timeliness. I want the data to be at most 5 minutes old. So this is a near-real-time system.
2. Comprehensiveness. No sampling. Complete data sets.
3. Accuracy (how precise the data is). Everything should be accurate.
4. Accessibility. Getting to meaningful data in Google Analytics is awful. To start with it’s all 12 - 24 hours old, and this is a huge impediment to insight & action.
5. Performance. Most reports / dashboards should render in under 5 seconds.
6. Durability. Keep all stats for all time. I know this can get rather tough, but it’s just text.

The 23-year-old programmer inside of me is salivating at the idea of building this. The burned out 27-year-old programmer inside of me is busy writing an email about how all of these demands, taken together, probably violate the CAP theorem somehow and also, hey, did you know that accuracy and precision are different?

But the 33-year-old programmer (who has long since beaten those demons into a bloody submission) sees the difficulty as irrelevant at best. Real-time analytics are undesirable. While there are many things wrong with our infrastructure, I would argue that the waiting is not one of those things.

Engineers might find this assertion more puzzling than most. I am sympathetic to this mindset, and I can understand why engineers are predisposed to see instantaneous A/B statistics as self-evidently positive. We monitor everything about our site in real time. Real-time metrics and graphing are the key to deploying 40 times daily with relative impunity. Measure anything, measure everything!

Part of the deploy dashboard at Etsy. We love up-to-the-minute graphs.

This line of thinking is a trap. It’s important to divorce the concepts of operational metrics and product analytics. Confusing how we do things with how we decide which things to do is a fatal mistake.

So what is it that makes product analysis different? Well, there are many ways to screw yourself with real-time analytics. I will endeavor to list a few.

The first and most fundamental way is to disregard statistical significance testing entirely. This is a rookie mistake, but it’s one that’s made all of the time. Let’s say you’re testing a text change for a link on your website. Being an impatient person, you decide to do this over the course of an hour. You observe that 20 people in bucket A clicked, but 30 in bucket B clicked. Satisfied, and eager to move on, you choose bucket B. There are probably thousands of people doing this right now, and they’re getting away with it.

This is a mistake because there’s no measurement of how likely it is that the observation (20 clicks vs. 30 clicks) was due to chance. Suppose that we weren’t measuring text on hyperlinks, but instead we were measuring two quarters to see if there was any difference between the two when flipped. As we flip, we could see a large gap between the number of heads received with either quarter. But since we’re talking about quarters, it’s more natural to suspect that that difference might be due to chance. Significance testing lets us ascertain how likely it is that this is the case.

A subtler error is to do significance testing, but to halt the experiment as soon as significance is measured. This is always a bad idea, and the problem is exacerbated by trying to make decisions far too quickly. Funny business with timeframes can coerce most A/B tests into statistical significance.

A simulation of flipping two fair coins. In the green regions, the difference in the number of heads is measured to be significant. If we stopped flipping in those regions, we would (incorrectly) decide the coins were different.

Depending on the change that’s being made, making any decision based on a single day of data could be ill-conceived. Even if you think you have plenty of data, it’s not farfetched to imagine that user behavior has its own rhythms. A conspicuous (if basic) example of this is that Etsy sees 30% more orders on Tuesdays than it does on Sundays.

Gratuitous infographic courtesy Brendan Sudol

While the sale count itself might not skew a random test, user demographics could be different day over day. Or very likely, you could see a major difference in user behavior immediately upon releasing a change, only to watch it evaporate as users learn to use new functionality. Given all of these concerns, the conservative and reasonable stance is to only consider tests that last a few days or more.

One could certainly have a real-time analytics system without making any of these mistakes. (To be clear, I find this unlikely. Idle hands stoked by a stream of numbers are the devil’s playthings.) But unless the intention is to make decisions with this data, one might wonder what the purpose of such a system could possibly be. Wasting the effort to erect complexity for which there is no use case is perhaps the worst of all of these possible pitfalls.

For all of these reasons I’ve come to view delayed analytics as positive. The turnaround time also imposes a welcome pressure on experimental design. People are more likely to think carefully about how their controls work and how they set up their measurements when there’s no promise of immediate feedback.

Real-time web analytics is a seductive concept. It appeals to our desire for instant gratification. But the truth is that there are very few product decisions that can be made in real time, if there are any at all. Analysis is difficult enough already, without attempting to do it at speed.

You most likely don't need metrics

Mike's Notes

Below is an article by Damien Mathieu on observability. I don't know what I will need to do here yet. To start, Ajabbi will need a hybrid cloud to host its database and web servers. Pipi has internal logs in place, which it uses as feedback.

Resources

• https://dmathieu.com/en/opinions/no-need-for-metrics/
• https://opentelemetry.io/
• https://en.wikipedia.org/wiki/Column-oriented_DBMS
• https://youtu.be/tr2KcekX2kk?si=iUQMrgcUu-cc6eil
• https://www.usenix.org/conference/nsdi20/presentation/hauer

Repository

• Home > Ajabbi Research > Library

Last Updated

18/03/2025

You most likely don't need metrics

By: Damien Mathieu

dmathieu.com: September 5 2022

Ever since we need to operate hardware and software in production, we have needed to know how those behave. For example, when I brew craft beer, I use an iSpindle to monitor the temperature and the gravity of my wort.

Every 15 minutes, the tool wakes up, checks the value it has to check, and sends them to a provider which stores that data and shows me graphs.

This works very well, and allows good monitoring of production impact. If my fridge stops working and my wort start going into too high temperatures, I can be alerted and fix the problem.

But does this also work well for modern web platforms?

The limits of metrics

The main difference between my small home brew setup and a web platform is the scale. While I’m sending metrics once every 15 minutes, your platform will have thousands of requests every second (or at least, that’s the hope).

A web platform will also need to track way more data than I need with my fermentations. There will be user agents, IP addresses, user IDs, and much more that you need to know about within your observability data to be able to properly investigate issues.

Cardinality

The cardinality of a metric is a number of individual values it has. If you have a metric for HTTP requests which has the HTTP method as an attribute, and the possible values are GET and POST, your cardinality is 2. If you add the status code as an attribute, and you can have 200 and 500 as status codes, then your cardinality is 4. Now, if you want to know the latency of HTTP requests for a specific customer, you need to add their ID as an attribute, which will up your cardinality to your number of customers, multiplied by the cardinality of all the other attributes. Indeed, each metric with different attributes is considered as a different metric.

Unless you want to overwhelm your Time Series Database, you do not want high cardinality in your metrics. Tracking the HTTP method is fine. Tracking the major number of the status code (for example, 2xx) is fine too. But don’t put a user ID in there, or your bill will explode quite quickly.

As a general rule, all your metric attributes should have bounds, so you can predict your cardinality, and the scale of your TSDB. Setting any unbounded attributes (such as user ID) will give you issues at scale.

Aggregates

If low cardinality were the only issue with metrics, that wouldn’t be so problematic. We could “just” scale up our databases to handle more data, or write new engines that allow higher cardinality. But to emit metrics, your service is also aggregating data, losing context which can be very important when investigating an issue.

Metrics work well to measure counters or gauges, which are values that go up or down but represent a single event. For example, CPU usage. However, they have a major issue when measuring histograms, such as latency, or request size.

In OpenMetrics format, metrics measuring the latency of an HTTP service would be emitted like this:

http_request_duration_seconds_sum{service="api"} 8953.332

http_request_duration_seconds_count{service="api"} 38526

http_request_duration_seconds_bucket{service="api", le="0"}

http_request_duration_seconds_bucket{service="api", le="0.01"} 0

http_request_duration_seconds_bucket{service="api", le="0.025"} 8

http_request_duration_seconds_bucket{service="api", le="0.05"} 1672

http_request_duration_seconds_bucket{service="api", le="0.1"} 8954

http_request_duration_seconds_bucket{service="api", le="+Inf"} 27892

This histogram has several metrics:

• The _sum metric is the sum of all measurements.
• The _count metric is the total number of HTTP requests being made.

The rest of the metrics, in _bucket are the number of HTTP requests being made and for which the response time is lower or equal to the value tagged, and higher than the smaller bucket value.

For example, we know we have 8 requests for which the duration was between 0.01 and 0.025ms. We also know that 27892 requests had a response time higher than 0.1ms. However, we have no way of knowing whether the requests in the first bucket were at 0.02ms, or at 0.024ms. Or if the ones in the second were at 0.2ms, or 30s.

Of course, every metrics client will allow you to set up the buckets of your choice, allowing you to set much higher values than 0.1. But if an unexpected event happens and your metrics aren’t (yet) properly configured to track it, you will have no way of properly investigating it.

Unknown Unknowns

To properly operate a service in production, you need to be able to know what is happening within it, starting at the macro level, and being able to dig deep into specific requests. As we have seen, metrics only allow you to look at the macro level. They don’t allow you to dig into specific problematic requests.

If you know what the problem is, and just need to know when it happens, they are very good. However, within all our modern web services, problems happen that we just don’t know about. And we especially need to know about those.

If you get alerted for high HTTP latency on a service, it could be a global issue, impacting all the service’s requests. But it could also be a specific issue impacting only a specific endpoint, or a specific subset of customers. If you can’t answer that question, all you’re doing during incidents is guessing, using tribal knowledge (and promoting hero culture).

We can do better

OK, so what do we want? We want a system where we can send (very) high cardinality data. We want a system where we can gather macro-level graphs, and then dig into specific requests.

You may already have thought of (semantic) logs, and you would be right! Let’s say with each HTTP request, I’m sending the following JSON log line:

{

"method": "POST",

"path": "/users",

"user_id": "b38239dd-aebc-4a41-8ed9-55f7ff7d77bb",

"request_id": "a61a989a-2fc6-483a-965d-cc706d80061b",

"user_agent": "Mozilla/5.0 [...]",

"started_at": 1662105599349,

"finished_at": 1662105599352,

"status_code": 200

}

Then, when the log is ingested, the database can also aggregate some pre-set metrics that will be used for alerting. The day you need to investigate an issue on the service, you can start with the computed metric and easily switch to micro-level investigations directly on the requests themselves.

The next iteration

Logs are an awesome first step towards observability. But they lack context. Our log line above has a request_id field that we can reuse in other log entries to get everything that happened within a specific request. But that view is going to be hard to read as it is linear. If your platform has several services, it is also going to be hard to link two (or more) requests together.

Now, what we need is convention for our logs. We want several things:

A way to propagate our request IDs across services.

A conventional way to pass start/end times, so our visualization service can better display our log entries.

By now, you may see where I’m going. So let’s give it: if you don’t need metrics, you also don’t need logs. What you want is traces.

Take the following Go example using OpenTelemetry:

func httpHandler(w http.ResponseWriter, r *http.Request) {

ctx, span := tracer.Start(r.Context(), "hello-span")

defer span.End()

w.Write([]byte("Hello World"))

}

This bit of code will emit a span (which again, can be seen as a log with lots of conventions) tied to the function. Once sent to your provider, it can be displayed as a gantt chart, to view exactly what happened during a request.

Cardinality

Your tracing provider will likely use a column-based datastore, where data will be stored in columns.

Storing the traces in columns allows quickly retrieving data, even if they have very different attributes, and no matter how many they have. So you can have spans with hundreds of attributes, and contrarily to metrics, it won’t make your database vastly bigger nor slower.

For more details on how column-based datastores for traces work, I recommend this talk:

Aggregates

With traces, you are now sending raw data (though possibly sampled). So it is not aggregated anymore. You can easily track the exact latency outliers, since you’re retaining the exact attributes stored from them.

Of course, data may be aggregated within your provider, to build graphs that provide a longer data retention than your raw span data is kept. But those graphs don’t impact your service’s code, and your recent raw data is always available to be queried and to investigate.

And they lived happily ever after

With this way of storing your observability data, you can be paged for an increased HTTP latency issue, see that only a specific endpoint is impacted by that latency. And even that only a few customers are facing the issue. You can even know if a specific SQL query is causing contention and the latency. With that information, you can properly mitigate the issue, and better communicate it with your customers.

When metrics only offer you an approximation of what’s happening in production, traces will allow you to scientifically validate the root cause of an issue.

While metrics require intrinsic knowledge of a service to be able to properly understand issues, by ditching them and relying on traces instead, you can shift your entire on-call culture. People can query their traces for any question they might have. And the person who can best debug an issue won’t be the most senior in the team anymore, but the most curious one.

Need more convincing?

By this point, you may be thinking I’m just being pedantic and unrealistic, in which case I would just say that it’s not because something worked great for years that we shouldn’t reconsider it from scratch if something better is coming up.

If you want to try it out by yourself, take a look at the OpenTelemetry demo.

Metrics are still sometimes useful

Most of the metrics we use today should disappear. For example, there’s no need for a metric with HTTP error rates anymore. You can infer that from traces. However, there will still be times when metrics are useful. My iSpindle example from the beginning of this article is a good case.

Even a web service can benefit from some metrics, even though you may realize they should be quite rare. Anything that cannot be encompassed in an event (an HTTP request is an event) but still needs to be tracked is a good candidate for a metric. For example, tracking available disk space for a database is a good metric. And it will allow you to be aware of upcoming issues much earlier than waiting for errors being raised due to missing disk space.

Conclusion

When the Google SRE book was released in 2016, it started a small revolution, and suddenly, everybody wanted to create SLOs. But nobody really had the tooling to do them. We were sold on creating error budgets, but couldn’t do them with metrics, as they weren’t granular enough. Now, the market has slowly started to shift to allow us to really apply the principles outlined in SRE, and further, get meaningful availability.

Traces are a great way to achieve that increased observability on our platforms, and finally be able to really know what’s happening on them, and not just guess.