Reduce Datadog Costs: What a Fintech on AWS Can Actually Cut

Every fintech we work with runs Datadog, and every one has had the meeting where the bill doubled and nobody could say why. The usual first move is to cut log volume at the source. It is usually the wrong move: at list price, ingestion is the cheapest line on the invoice. Datadog costs come from three other places, each a configuration decision nobody made on purpose.

The regulated version is worse, because the reason given for indexing everything for 30 days is "the auditor". The RBI and CERT-In want logs retained, in India, for a long time. Neither wants them searchable in a SaaS console in Virginia. Separate the compliance copy from the debugging copy and most of the argument for the big index disappears.

Our position: reduce Datadog costs by fixing three settings, an S3 archive in Mumbai, exclusion filters with a daily quota, and tag configuration on your custom metrics, before you touch a single log line in application code.

Where does a Datadog bill actually come from?

From units that are easy to confuse. Datadog's billing page lists them: hosts, containers, custom metrics, APM hosts, ingested and indexed spans, ingested log gigabytes and indexed log events. The ones that grow without anyone deciding are indexed events, custom metrics, containers and the host count. Ingested gigabytes, the thing people cut first, are rarely the problem.

The invoice arrives as one number, and the console shows usage, not cost. So the first thing we do in any account is put the estimated-usage metrics on a dashboard: datadog.estimated_usage.logs.ingested_events, datadog.estimated_usage.metrics.custom.by_metric and datadog.estimated_usage.hosts. Datadog documents them and warns they run 10 to 20 percent off billable usage. That is close enough to find the line that doubled.

Why does indexing cost seventeen times what ingestion does?

Take the list prices as published: ingestion is $0.10 per gigabyte, and standard indexing at 15 days is $1.70 per million events, $2.55 on demand. Now take a platform that produces 100 gigabytes of logs a day at roughly a kilobyte per event. That is 3,000 gigabytes and 3 billion events a month. Ingesting all of it costs $300. Indexing all of it costs $5,100, and $7,650 if you are over commitment. Cutting log volume at the source shaves the $300 line. Deciding what to index moves the $5,100 one.

Two Datadog features act on the expensive line and leave the cheap one alone. Exclusion filters on an index drop matching logs from indexing, with a sample rate, and the documentation is explicit that "excluded logs are discarded from indexes, but still flow through the Livetail and can be used to generate metrics and archived". A daily quota on the index is the hard stop: once reached, logs "are no longer indexed but are still available in the livetail, sent to your archives, and used to generate metrics". So the health-check noise and the debug-level chatter from a service someone forgot to turn down still arrive, still hit the archive, still feed metrics, and stop costing $1.70 a million.

Flex Logs is the middle tier for logs you might need in three months but will not alert on. Storage is $0.05 per million events, plus a flat-rate compute tier sized to how much you query, which is a different order of magnitude from standard indexing, with the trade that monitors are not supported. That is fine for audit trails. It is not fine for the logs your on-call pages on.

Which logs does the regulator want, and does Datadog need to hold them?

CERT-In's directions expect a rolling 180 days of logs, held within Indian jurisdiction; we covered that in the six-hour reporting post. Datadog's list of sites has the US, Germany, the UK, Japan and Australia on it. India is not there. So a fintech that keeps its only 180-day copy in a Datadog index is paying standard indexing rates for six months on a copy that does not satisfy the residency half of the requirement anyway.

The archive fixes both halves. A log archive forwards every ingested log, indexed or not, to a bucket you own; S3 is a supported destination. Put that bucket in ap-south-1 with a lifecycle rule to a colder storage class at 30 days and expiry at whatever your retention policy says, and the compliance copy now lives in India, on storage priced by the gigabyte, in a format you can read without Datadog. The Datadog index can then shrink to what on-call uses, in our experience 7 to 15 days. When someone needs March's logs, you rehydrate that window from the archive. You pay for the gigabytes scanned and for the events that land in the index, so keep the window narrow: "the query is applied after the files matching the time period are downloaded".

One caveat: the archive still passes through Datadog's site before it lands in your bucket. Whether that transit is acceptable is the analysis we described in the data localisation post, and for payment data the honest answer is usually to strip it from the log line before the agent sees it.

Why did one histogram blow through the custom metrics allowance?

Because a custom metric is not a metric name. Datadog's custom metrics billing page defines it as a metric name plus every unique set of tag values, host included, and then multiplies: a gauge is one custom metric per tag combination, a histogram is five, a distribution is five, or ten with percentiles. Pro includes 100 per host, pooled across the account.

So twenty hosts on Pro gives you 2,000. A developer adds one histogram, http.request.latency, tagged by endpoint and status. Fifty endpoints, five status codes, twenty hosts, five aggregations: 25,000 custom metrics from one line of code. The overage is billed per 100 metrics at your contract rate, and nobody sees it until the invoice. We have watched the same shape with a user id tag, which is unbounded and turns the number into your monthly active users.

Metrics without Limits is the fix that does not require a code change. You set an allowlist of queryable tags per metric, and Datadog indexes only those combinations while still ingesting the rest. Drop host from the latency histogram and the 25,000 becomes 1,250. The console pre-fills the allowlist from tags queried in the last 30 days, which is a good way to discover that nobody has ever looked at the metric.

What does the ninth-busiest hour have to do with your host bill?

Infrastructure and APM hosts are billed on a high-water mark. Datadog records the host count every hour, sorts the month from highest to lowest and, per the APM billing page, "charges based on the ninth highest measurement". The top eight hours are forgiven. The ninth sets the bill.

For a fintech on AWS with autoscaling, that means one event that adds twenty hosts for nine hours, a settlement run, a marketing push, a batch job that fell behind, prices the whole month as if they were always there. Containers follow the same logic: Pro includes 5 per host, Enterprise 10, and anything above that is billed per container-hour. A Kubernetes cluster with small nodes and many pods is over the allotment on day one.

The fix is not to stop scaling. It is to know which scale-outs need the agent. A short-lived batch fleet that produces CloudWatch metrics anyway may not need a Datadog host licence for nine hours.

What can you change on Monday?

The index is the biggest lever, and it is one Terraform resource. This is the provider's example adapted to a production index that keeps 7 days in standard tier, 180 in Flex, hard-stops at a daily quota and drops the two noisiest sources:

resource "datadog_logs_index" "production" {
  name                = "production"
  daily_limit         = 100000000
  daily_limit_warning_threshold_percentage = 70
  retention_days      = 7
  flex_retention_days = 180
  filter { query = "env:production" }

  exclusion_filter {
    name       = "ALB health checks"
    is_enabled = true
    filter {
      query       = "source:elb @http.url_details.path:/healthz"
      sample_rate = 1.0
    }
  }
  exclusion_filter {
    name       = "Debug level"
    is_enabled = true
    filter {
      query       = "status:debug"
      sample_rate = 0.95
    }
  }
}

Before applying it, run the two exclusion queries in the Log Explorer over the last seven days and write down the event counts. That number times $1.70 per million is the monthly saving, and it is the figure that gets the change approved. Create the S3 archive in ap-south-1 in the same Terraform run, so the index shrink and the compliance copy land together.

When is the bigger bill the right answer?

Sometimes. A payments team in the middle of a fraud investigation needs 30 days searchable, and rehydrating from S3 every morning is slower than paying for the index. A team with three engineers and no platform function yet is better off overpaying Datadog than running its own Loki and Prometheus, because the engineer-months that stack costs are worth more than the invoice. And if the most expensive custom metric is the one the risk team watches for settlement failures, its cardinality is the point, not the problem.

The test is whether anyone can say what each expensive line is for. If the answer is "the auditor", check the archive. If it is "we have always done it that way", that is the line to cut.

The short version

Put the estimated-usage metrics on one dashboard so the bill has line items. Create the S3 archive in Mumbai first, so the 180-day compliance copy exists before anything is shortened. Then shrink the standard index to what on-call uses, put the noisiest sources behind exclusion filters, set a daily quota with a warning at 70 percent, and move audit-trail logs to Flex. Sort the custom metrics page by count and put an allowlist on every histogram and distribution that carries host or a user identifier. Look at last month's ninth-busiest hour and ask whether those hosts needed an agent.

None of that removes a log or a metric your regulator or your on-call needs. It removes the ones nobody chose. If you cannot say which lines on the invoice those are, that is the audit.