In Asp.Net Core Can't export traces from OpenTelemetry exporter to elastic apm

Question

Recently, I found OpenTelemetry, and I like to use it in my application. First, I created a new Asp.Net Core 5 web API and exported traces on the console. After that, I tried OpenTelemetry.Exporter.OpenTelemetryProtocol for export traces to Elastic APM, but I couldn't see log or traces at elastic. In 3rd step, I used Elastic.Apm.NetCoreAll, and I successfully sent traces to APM. So I think my Elastic APM configurations are correct, but some configurations at asp are wrong.

Can anyone help me to find a mistake?

EDIT 1: I use:
elastic search self-managed 8.0.0
elastic APM 8.0.0 integration (but I don't 100% sure because in Observability > APM > Settings > Schema Show APM Server binary currently settup and elastic agent is disable.

My startup:

var serviceName = "apm_testElastic";
var serviceVersion = "1.0.0";
services.AddOpenTelemetryTracing(traceProviderBuilder =>
        {
            traceProviderBuilder
            .AddSource(serviceName)
            .SetResourceBuilder(
                ResourceBuilder.CreateDefault()
                .AddService(serviceName: serviceName, serviceVersion: serviceVersion)
                .AddEnvironmentVariableDetector())
            .AddAspNetCoreInstrumentation(option => { option.RecordException = true; })
            .AddHttpClientInstrumentation(option => { option.RecordException = true; })
            .AddSqlClientInstrumentation()
            .AddConsoleExporter()
            .AddOtlpExporter(configs =>
            {
                configs.Endpoint = new Uri("http://localhost:8200");
                configs.Protocol = OpenTelemetry.Exporter.OtlpExportProtocol.HttpProtobuf;
            })
            ;
        });

Captured request that sent by OpenTelemetry.Exporter.OpenTelemetryProtocol:

POST http://localhost:8200/ HTTP/1.1
Host: localhost:8200
traceparent: 00-e5d530a09bdd276eac0b4721599eb9f3-2030a82060ad641b-00
Transfer-Encoding: chunked
Content-Type: application/x-protobuf

4F0

    
~
!
service.name
apm_testElastic

service.version 
1.0.0
=
service.instance.id&
$a98af9dc-0169-4077-9b6d-dbbdbc2b9018 
3
(OpenTelemetry.Instrumentation.AspNetCore 1.0.0.0 
    @   Y*r Z ^ ߁X */swagger/index.html09Ȣ^%L A   ?L J
    http.host
localhost:64385J

http.method
GETJ$

http.target
/swagger/index.htmlJ7
http.url+
)http://localhost:64385/swagger/index.htmlJc
http.user_agentP
NMozilla/5.0 (Windows NT 10.0; Win64; x64; rv:97.0) Gecko/20100101 Firefox/97.0J
http.status_code z  
  6 5
g 7[ e    J  s*/swagger/swagger-ui.css09 vHL A *$UL J
    http.host
localhost:64385J

http.method
GETJ(

http.target
/swagger/swagger-ui.cssJ;
http.url/
-http://localhost:64385/swagger/swagger-ui.cssJc
http.user_agentP
NMozilla/5.0 (Windows NT 10.0; Win64; x64; rv:97.0) Gecko/20100101 Firefox/97.0J
http.status_code z 
&
     M   <  D   d*/swagger/v1/swagger.json09t FXL A(T bL J
    http.host
localhost:64385J

http.method
GETJ)

http.target
/swagger/v1/swagger.jsonJ<
http.url0
.http://localhost:64385/swagger/v1/swagger.jsonJc
http.user_agentP
NMozilla/5.0 (Windows NT 10.0; Win64; x64; rv:97.0) Gecko/20100101 Firefox/97.0J
http.status_code z 
0

Captured Responces:

HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Tue, 22 Feb 2022 13:20:33 GMT
Content-Length: 132

{"build_date":"2022-02-03T16:50:23Z","build_sha":"6afb387ae002e5a8dc2427b53225aa42681e4208","publish_ready":true,"version":"8.0.0"}

EDIT 2:
captured request that sent by Elastic.Apm.NetCoreAll

POST http://localhost:8200/intake/v2/events HTTP/1.1
Host: localhost:8200
User-Agent: apm-agent-dotnet/1.14.0+a04553cf41c2bc212d866c6031f087273edb9a8b (apm_testElastic 1.0.0) System.Net.Http/5.0.721.25508 .NET_5/5.0.7
traceparent: 00-f7ac803fbe013847ab5179155097ac12-61b588fa5f3baf4c-00
Content-Type: application/x-ndjson; charset=utf-8
Content-Length: 7745

{"metadata":{"service":{"agent":{"name":"dotnet","version":"1.14.0+a04553cf41c2bc212d866c6031f087273edb9a8b"},"environment":"production","framework":{"name":"ASP.NET Core","version":"5.0.0.0"},"language":{"name":"C#"},"name":"apm_testElastic","node":{},"runtime":{"name":".NET 5","version":"5.0.7"},"version":"1.0.0"},"system":{"detected_hostname":"DESKTOP-VBBLGES","hostname":"DESKTOP-VBBLGES"}}}
{"transaction":{"context":{"request":{"body":"[REDACTED]","headers":{"Accept":"text/plain","Accept-Encoding":"gzip, deflate, br","Accept-Language":"en-US,en;q=0.5","Connection":"close","Cookie":"...","Host":"localhost:44388","Referer":"https://localhost:44388/swagger/index.html","Te":"trailers","User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:97.0) Gecko/20100101 Firefox/97.0","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin"},"http_version":"2","method":"GET","socket":{"remote_address":"127.0.0.1"},"url":{"full":"https://localhost:44388/WeatherForecast","hostname":"localhost","pathname":"/WeatherForecast","protocol":"HTTP","raw":"https://localhost:44388/WeatherForecast","search":""}},"response":{"finished":true,"headers":{"Content-Type":"application/json; charset=utf-8"},"status_code":200}},"duration":3537.739,"id":"b3b5abb9757d6c45","sampled":true,"name":"GET WeatherForecast/Get","outcome":"success","result":"HTTP 2xx","sample_rate":1.0,"span_count":{"dropped":0,"started":0},"timestamp":1645613137714025,"trace_id":"0abec8e78e6c4846886d04c8da27bf23","type":"request"}}

Two New Questions:
1- Two captured request format is different, is Elastic APM server handled both format?
2- Is the captured request from OpenTelemetry.Exporter.OpenTelemetryProtocol correct?

Answer 1

After some search, reinstall and debug I found my fault. I Downloaded AspNetCore sample from OpenTelemetry .net samples and I found that I should add next line in my code:

AppContext.SetSwitch("System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport", true);