Convert Prometheus Metrics to Open-Telemetry?

At work, we're implementing Open-Telemetry to monitor our applications and infrastructure. Mostly this is just a case of deploying the collectors and watching the signals flow in. At the moment I am focussing on getting metrics in from prometheus sources, e.g. applications, cAdvisor, and kube-state-metrics. Scraping them is easy using the prometheus receiver, getting them into a back-end is mostly trivial. What isn't obvious is if the metric names should be converted from prometheus format (thing_namespace_signal) to OTEL format (thing.namespace.signal). The guide for compatibility specifies that you SHOULD NOT modify the metric name, however the name is modified when the prometheus metrics are normalised into OTEL metrics. For example, in prometheus, if there is a histogram called http_request_duration_seconds, then there will metrics like

http_request_duration_seconds_bucket{le="0.05",method="POST",path="/api/auth-type",statusCode="200"} 1
http_request_duration_seconds_bucket{le="0.1",method="POST",path="/api/auth-type",statusCode="200"} 1
http_request_duration_seconds_bucket{le="0.5",method="POST",path="/api/auth-type",statusCode="200"} 2
http_request_duration_seconds_bucket{le="1",method="POST",path="/api/auth-type",statusCode="200"} 2
http_request_duration_seconds_bucket{le="2",method="POST",path="/api/auth-type",statusCode="200"} 2
http_request_duration_seconds_bucket{le="5",method="POST",path="/api/auth-type",statusCode="200"} 2
http_request_duration_seconds_bucket{le="+Inf",method="POST",path="/api/auth-type",statusCode="200"} 2
http_request_duration_seconds_sum{method="POST",path="/api/auth-type",statusCode="200"} 0.159817235
http_request_duration_seconds_count{method="POST",path="/api/auth-type",statusCode="200"} 2

As this histogram passes though the pipeline it becomes the following in the collector (using the debug exporter)

http_request_duration_seconds{method=POST,path=/api/auth-type,statusCode=200} count=2 sum=0.159817235 le0.05=1 le0.1=0 le0.5=1 le1=0 le2=0 le5=0 0

So as we can see the _bucket, _sum, and _count metrics have been flattened down into the original metric name. And we can also see that the histogram buckets have been flattened too. While this makes the metric more conscise it does show that the metric is being transformed on the way through the collector.

My question is if the prometheus receiver is normalising metrics based on the metric name why shouldn't they be further standardised into OTEL format?

The ulterior motive for this is that when we have metrics in a back-end (like Elasticsearch) then the namespace separator will be the same as all the other metrics meaning that end-users of the back-end are able to discover the metrics easier. For example the type-ahead search will offer results for k8s. before k8s_, or that end-users will need to know that metrics from one source.

One argument that I can see is that if you have metrics coming from say kube-state-metrics then you should really have the metric names be the same so that you can cross-reference between what is available in your back-end and the documentation.

However, while I was digging around writing this I stumbled on the OTTL Playground which provided me with an example of rewriting metric names using the transform processor. I'd not looked at the docs for the function replace_pattern() before now so went and had a look to check the behaviour. In the examples for the function is replace_pattern(metric.name, "^kube_([0-9A-Za-z]+_)", "k8s.$$1."), hold on that prefix is normally associated with metrics from Kubernetes, which implies that someone else has had the same idea.

From this we can create a processor block like the following that can be added into a metrics pipeline.

processor:
  transform:
    metric_statements:
    - context: metric
      statements:
      - replace_pattern(name, "_", ".")

so when the example histogram is ingested via a pipeline using this transform processor, it becomes

http.request.duration.seconds{method=POST,path=/api/auth-type,statusCode=200} count=2 sum=0.159817235 le0.05=1 le0.1=0 le0.5=1 le1=0 le2=0 le5=0 0

Result! We now have a metric that is consistent with the namespacing of the OTEL originated ones.