Vehicle Location using SpringBoot 3 with gRPC, OpenTelemetry and Micrometer Tracing — Part 1

Gabriel Martins
8 min readNov 21, 2023

Introduction

This article is an example of vehicle location using SpringBoot 3 with gRPC, OpenTelemetry and Micrometer Tracing.

It’s not a real case, but just an example to demonstrate how to use these tecnologies mentioned above.

The article will be divided in 3 parts:

  • Setup environment
  • Application Development
  • Tests and Usage

Setup Environment

In order to setup the development enviroment we need to create the following services:

OpenTelemetry Collector

Grafana

Grafana Tempo

Prometheus

Jaeger

Elasticsearch

Kibana

You can use the docker-compose.yaml file below

version: "3.8"
services:
# And put them in an OTEL collector pipeline...
otel-collector:
image: otel/opentelemetry-collector:0.72.0
command: [ "--config=/etc/otel-collector.yaml" ]
ports:
- "31888:1888" # pprof extension
- "8888:8888" # Prometheus metrics exposed by the collector
- "8889:8889" # Prometheus exporter metrics
- "13133:13133" # health_check extension
- "4317:4317" # OTLP gRPC receiver
- "4318:4318" # OTLP http receiver
- "55679:55679" # zpages extension
volumes:
- ./otel-collector.yaml:/etc/otel-collector.yaml

# To eventually offload to Tempo...
tempo:
image: grafana/tempo:2.2.4
command: [ "-config.file=/etc/tempo.yaml" ]
volumes:
- ./tempo.yaml:/etc/tempo.yaml
- ./tempo-data:/tmp/tempo
ports:
- "14268" # jaeger ingest
- "3200" # tempo
- "4317" # otlp grpc
- "4318" # otlp http
- "9411" # zipkin

prometheus:
image: prom/prometheus:v2.47.2
command:
- --config.file=/etc/prometheus.yaml
- --web.enable-remote-write-receiver
- --enable-feature=exemplar-storage
volumes:
- ./prometheus.yaml:/etc/prometheus.yaml
ports:
- "9090:9090"

jaeger-server:
image: jaegertracing/all-in-one:1.18.1
container_name: jaeger-server
ports:
- "5775:5775/udp"
- "6831:6831/udp"
- "5778:5778"
- "16686:16686"
- "14268:14268"
- "14250:14250"
- "9411:9411"

grafana:
image: grafana/grafana:10.2.0
volumes:
- ./grafana-datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml
environment:
- GF_AUTH_ANONYMOUS_ENABLED=true
- GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
- GF_AUTH_DISABLE_LOGIN_FORM=true
- GF_FEATURE_TOGGLES_ENABLE=traceqlEditor
ports:
- "3000:3000"
depends_on:
- otel-collector
- tempo
- prometheus
- jaeger-server

elasticsearch:
image: elasticsearch:8.10.4
container_name: elasticsearch
environment:
- node.name=es-node
- cluster.name=es-cluster
- discovery.type=single-node
- xpack.security.enabled=false
ports:
- "9200:9200"
- "9300:9300"
networks:
- elastic

kibana:
image: kibana:8.10.4
container_name: kibana
restart: "unless-stopped"
environment:
- "ELASTICSEARCH_URL=http://elasticsearch:9200"
ports:
- "5601:5601"
networks:
- elastic
depends_on:
- elasticsearch

networks:
elastic:

You will need these files too

prometheus.yaml

global:
scrape_interval: 15s
evaluation_interval: 15s

scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: [ 'host.docker.internal:9090' ]
- job_name: 'tempo'
static_configs:
- targets: [ 'tempo:3200' ]
- job_name: 'sensor-service'
metrics_path: '/actuator/prometheus'
scrape_interval: 3s
static_configs:
- targets: ['host.docker.internal:8080']
labels:
application: 'Sensor Service'
- job_name: 'vehicle-service'
metrics_path: '/actuator/prometheus'
scrape_interval: 3s
static_configs:
- targets: ['host.docker.internal:8082']
labels:
application: 'Vehicle Service'

otel-collector.yaml

receivers:
otlp:
protocols:
grpc:
http:

processors:
batch:

exporters:
otlp:
endpoint: tempo:4317
tls:
insecure: true
jaeger:
endpoint: jaeger-server:14250
tls:
insecure: true

service:
pipelines:
traces:
receivers: [ otlp ]
processors: [ batch ]
exporters: [ otlp, jaeger ]

tempo.yaml

server:
http_listen_port: 3200

query_frontend:
search:
duration_slo: 5s
throughput_bytes_slo: 1.073741824e+09
trace_by_id:
duration_slo: 5s

distributor:
receivers: # this configuration will listen on all ports and protocols that tempo is capable of.
jaeger: # the receives all come from the OpenTelemetry collector. more configuration information can
protocols: # be found there: https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver
thrift_http: #
grpc: # for a production deployment you should only enable the receivers you need!
thrift_binary:
thrift_compact:
zipkin:
otlp:
protocols:
http:
grpc:
opencensus:

ingester:
max_block_duration: 5m # cut the headblock when this much time passes. this is being set for demo purposes and should probably be left alone normally

compactor:
compaction:
block_retention: 1h # overall Tempo trace retention. set for demo purposes

metrics_generator:
registry:
external_labels:
source: tempo
cluster: docker-compose
processor:
span_metrics:
dimensions: ['correlation_id', 'start_time_ms', 'end_time_ms']
traces_storage:
path: /tmp/tempo/generator/traces
storage:
path: /tmp/tempo/generator/wal
remote_write:
- url: http://prometheus:9090/api/v1/write
name: prometheus_remote_write_tempo
send_exemplars: true

storage:
trace:
backend: local # backend configuration to use
wal:
path: /tmp/tempo/wal # where to store the the wal locally
local:
path: /tmp/tempo/traces

overrides:
metrics_generator_processors: [ 'local-blocks', service-graphs, span-metrics ]

grafana-datasources.yaml

apiVersion: 1

datasources:
- name: Prometheus
type: prometheus
uid: prometheus
access: proxy
orgId: 1
url: http://prometheus:9090
basicAuth: false
isDefault: false
version: 1
editable: false
jsonData:
httpMethod: GET
- name: Tempo
type: tempo
access: proxy
orgId: 1
url: http://tempo:3200
basicAuth: false
isDefault: true
version: 1
editable: false
apiVersion: 1
uid: tempo
jsonData:
httpMethod: GET
serviceMap:
datasourceUid: prometheus

After that let’s create the applications sensor-service and location-service

For the sensor service you can use the following build.gradle

buildscript {
apply from: 'repositories.gradle', to: buildscript
ext {
springBootVersion = '3.1.5'
springDependencyManagamentVersion = '1.1.3'
protobufPluginVersion = '0.9.4'
}
}

plugins {
id'java'
id 'org.springframework.boot' version "${springBootVersion}"
id 'io.spring.dependency-management' version "${springDependencyManagamentVersion}"
id 'com.google.protobuf' version "${protobufPluginVersion}"
id 'idea'
id 'eclipse'
}

group = 'br.gasmartins.sensors'
version = '0.0.1-SNAPSHOT'

java {
sourceCompatibility JavaVersion.VERSION_17
targetCompatibility JavaVersion.VERSION_17
}

configurations {
compileOnly {
extendsFrom annotationProcessor
}
}

apply from: 'repositories.gradle'

ext {
set('snippetsDir', file("build/generated-snippets"))
set('springBootVersion', "3.1.5")
set('springCloudVersion', "2022.0.4")
set('springGrpcVersion', "2.15.0.RELEASE")
set('grpcVersion', "1.59.0")
set('protobufVersion', "3.25.0")
set('annotationsApiVersion', "6.0.53")
set('googleProtoVersion', "2.28.0")
set('logstashVersion', "7.4")
set('otelGrpcVersion', "1.31.0-alpha")
set('otelLogsVersion', "1.26.0-alpha")
set('instancioVersion', "3.5.1")
set('awaitilityVersion', "4.2.0")
}

dependencies {
implementation 'org.springframework.boot:spring-boot-starter'
implementation 'org.springframework.boot:spring-boot-starter-actuator'
implementation 'org.springframework.boot:spring-boot-starter-data-elasticsearch'

implementation "net.devh:grpc-server-spring-boot-starter:${springGrpcVersion}"
implementation "net.devh:grpc-client-spring-boot-starter:${springGrpcVersion}"

implementation 'org.springframework.boot:spring-boot-starter-web'

compileOnly 'org.projectlombok:lombok'
annotationProcessor 'org.projectlombok:lombok'
testCompileOnly 'org.projectlombok:lombok'
testAnnotationProcessor 'org.projectlombok:lombok'

implementation 'io.micrometer:micrometer-tracing'
implementation 'io.micrometer:micrometer-tracing-bridge-otel'
runtimeOnly 'io.micrometer:micrometer-registry-prometheus'

implementation "io.opentelemetry:opentelemetry-exporter-otlp"
implementation "io.opentelemetry.instrumentation:opentelemetry-grpc-1.6:${otelGrpcVersion}"
implementation "io.opentelemetry:opentelemetry-api-logs:${otelLogsVersion}"

developmentOnly 'org.springframework.boot:spring-boot-devtools'
annotationProcessor 'org.springframework.boot:spring-boot-configuration-processor'

implementation "net.logstash.logback:logstash-logback-encoder:${logstashVersion}"

implementation 'io.grpc:grpc-auth'
implementation 'io.grpc:grpc-alts'
implementation 'io.grpc:grpc-netty'
implementation 'io.grpc:grpc-netty-shaded'
implementation 'io.grpc:grpc-googleapis'
implementation 'io.grpc:grpc-protobuf'
implementation 'io.grpc:grpc-stub'
implementation 'com.google.protobuf:protobuf-java'
implementation 'com.google.protobuf:protobuf-java-util'
implementation "com.google.api.grpc:proto-google-common-protos:${googleProtoVersion}"

if (JavaVersion.current().isJava9Compatible()) {
// Workaround for @javax.annotation.Generated
// see: https://github.com/grpc/grpc-java/issues/3633
compileOnly "org.apache.tomcat:annotations-api:${annotationsApiVersion}"
annotationProcessor "jakarta.annotation:jakarta.annotation-api"
}

testImplementation 'org.springframework.boot:spring-boot-starter-test'
testImplementation 'org.springframework.boot:spring-boot-testcontainers'
testImplementation 'org.testcontainers:elasticsearch'
testImplementation 'org.testcontainers:junit-jupiter'
testImplementation 'io.grpc:grpc-testing'
testImplementation "org.instancio:instancio-junit:${instancioVersion}"
testImplementation "org.awaitility:awaitility:${awaitilityVersion}"
}

dependencyManagement {
imports {
mavenBom "org.springframework.boot:spring-boot-dependencies:${springBootVersion}"
mavenBom "org.springframework.cloud:spring-cloud-dependencies:${springCloudVersion}"
mavenBom "io.grpc:grpc-bom:${grpcVersion}"
mavenBom "com.google.protobuf:protobuf-bom:${protobufVersion}"
}
}

tasks.named('test') {
outputs.dir snippetsDir
useJUnitPlatform()
}

tasks.named('asciidoctor') {
inputs.dir snippetsDir
dependsOn test
}

protobuf {
protoc {
artifact = "com.google.protobuf:protoc:${protobufVersion}"
}
plugins {
grpc {
artifact = "io.grpc:protoc-gen-grpc-java:${grpcVersion}"
}
}
generateProtoTasks {
all()*.plugins {
grpc {

}
}
}
}

sourceSets {
main {
java {
srcDirs 'build/generated/source/proto/main/grpc'
srcDirs 'build/generated/source/proto/main/java'
}
}
}

// Optional
eclipse {
classpath {
file.beforeMerged { cp ->
def generatedGrpcFolder = new org.gradle.plugins.ide.eclipse.model.SourceFolder('src/generated/main/grpc', null);
generatedGrpcFolder.entryAttributes['ignore_optional_problems'] = 'true';
cp.entries.add( generatedGrpcFolder );
def generatedJavaFolder = new org.gradle.plugins.ide.eclipse.model.SourceFolder('src/generated/main/java', null);
generatedJavaFolder.entryAttributes['ignore_optional_problems'] = 'true';
cp.entries.add( generatedJavaFolder );
}
}
}

// Optional
idea {
module {
sourceDirs += file("src/generated/main/java")
sourceDirs += file("src/generated/main/grpc")
generatedSourceDirs += file("src/generated/main/java")
generatedSourceDirs += file("src/generated/main/grpc")
}
}

To the location service you can use the follow build.gradle

buildscript {
apply from: 'repositories.gradle', to: buildscript
ext {
springBootVersion = '3.1.5'
springDependencyManagamentVersion = '1.1.3'
protobufPluginVersion = '0.9.4'
}
}

plugins {
id'java'
id 'org.springframework.boot' version "${springBootVersion}"
id 'io.spring.dependency-management' version "${springDependencyManagamentVersion}"
id 'com.google.protobuf' version "${protobufPluginVersion}"
id 'idea'
id 'eclipse'
}

group = 'br.gasmartins.locations'
version = '0.0.1-SNAPSHOT'

java {
sourceCompatibility JavaVersion.VERSION_17
targetCompatibility JavaVersion.VERSION_17
}

configurations {
compileOnly {
extendsFrom annotationProcessor
}
}

apply from: 'repositories.gradle'

ext {
set('snippetsDir', file("build/generated-snippets"))
set('springBootVersion', "3.1.5")
set('springCloudVersion', "2022.0.4")
set('springGrpcVersion', "2.15.0.RELEASE")
set('feignOkHttpVersion', "13.1")
set('grpcVersion', "1.59.0")
set('protobufVersion', "3.25.0")
set('annotationsApiVersion', "6.0.53")
set('googleProtoVersion', "2.28.0")
set('logstashVersion', "7.4")
set('otelGrpcVersion', "1.31.0-alpha")
set('otelLogsVersion', "1.26.0-alpha")
set('instancioVersion', "3.5.1")
}

dependencies {
implementation 'org.springframework.boot:spring-boot-starter'
implementation 'org.springframework.boot:spring-boot-starter-actuator'

implementation "net.devh:grpc-server-spring-boot-starter:${springGrpcVersion}"
implementation "net.devh:grpc-client-spring-boot-starter:${springGrpcVersion}"

implementation 'org.springframework.boot:spring-boot-starter-web'

implementation 'org.springframework.cloud:spring-cloud-starter-bootstrap'
implementation 'org.springframework.cloud:spring-cloud-starter-openfeign'
implementation "io.github.openfeign:feign-core:${feignOkHttpVersion}"
implementation "io.github.openfeign:feign-okhttp:${feignOkHttpVersion}"
implementation "io.github.openfeign:feign-micrometer:${feignOkHttpVersion}"

compileOnly 'org.projectlombok:lombok'
annotationProcessor 'org.projectlombok:lombok'
testCompileOnly 'org.projectlombok:lombok'
testAnnotationProcessor 'org.projectlombok:lombok'

implementation 'io.micrometer:micrometer-tracing'
implementation 'io.micrometer:micrometer-tracing-bridge-otel'
runtimeOnly 'io.micrometer:micrometer-registry-prometheus'

implementation "io.opentelemetry:opentelemetry-exporter-otlp"
implementation "io.opentelemetry.instrumentation:opentelemetry-grpc-1.6:${otelGrpcVersion}"
implementation "io.opentelemetry:opentelemetry-api-logs:${otelLogsVersion}"

developmentOnly 'org.springframework.boot:spring-boot-devtools'
annotationProcessor 'org.springframework.boot:spring-boot-configuration-processor'

implementation "net.logstash.logback:logstash-logback-encoder:${logstashVersion}"

implementation 'io.grpc:grpc-auth'
implementation 'io.grpc:grpc-alts'
implementation 'io.grpc:grpc-netty'
implementation 'io.grpc:grpc-netty-shaded'
implementation 'io.grpc:grpc-googleapis'
implementation 'io.grpc:grpc-protobuf'
implementation 'io.grpc:grpc-stub'
implementation 'com.google.protobuf:protobuf-java'
implementation 'com.google.protobuf:protobuf-java-util'
implementation "com.google.api.grpc:proto-google-common-protos:${googleProtoVersion}"

if (JavaVersion.current().isJava9Compatible()) {
// Workaround for @javax.annotation.Generated
// see: https://github.com/grpc/grpc-java/issues/3633
compileOnly "org.apache.tomcat:annotations-api:${annotationsApiVersion}"
annotationProcessor "jakarta.annotation:jakarta.annotation-api"
}

testImplementation 'org.springframework.boot:spring-boot-starter-test'
testImplementation 'org.springframework.boot:spring-boot-testcontainers'

testImplementation 'io.grpc:grpc-testing'
testImplementation "org.instancio:instancio-junit:${instancioVersion}"
}

dependencyManagement {
imports {
mavenBom "org.springframework.boot:spring-boot-dependencies:${springBootVersion}"
mavenBom "org.springframework.cloud:spring-cloud-dependencies:${springCloudVersion}"
mavenBom "io.grpc:grpc-bom:${grpcVersion}"
mavenBom "com.google.protobuf:protobuf-bom:${protobufVersion}"
}
}

tasks.named('test') {
outputs.dir snippetsDir
useJUnitPlatform()
}

protobuf {
protoc {
artifact = "com.google.protobuf:protoc:${protobufVersion}"
}
plugins {
grpc {
artifact = "io.grpc:protoc-gen-grpc-java:${grpcVersion}"
}
}
generateProtoTasks {
all()*.plugins {
grpc {

}
}
}
}

sourceSets {
main {
java {
srcDirs 'build/generated/source/proto/main/grpc'
srcDirs 'build/generated/source/proto/main/java'
}
}
}

// Optional
eclipse {
classpath {
file.beforeMerged { cp ->
def generatedGrpcFolder = new org.gradle.plugins.ide.eclipse.model.SourceFolder('src/generated/main/grpc', null);
generatedGrpcFolder.entryAttributes['ignore_optional_problems'] = 'true';
cp.entries.add( generatedGrpcFolder );
def generatedJavaFolder = new org.gradle.plugins.ide.eclipse.model.SourceFolder('src/generated/main/java', null);
generatedJavaFolder.entryAttributes['ignore_optional_problems'] = 'true';
cp.entries.add( generatedJavaFolder );
}
}
}

// Optional
idea {
module {
sourceDirs += file("src/generated/main/java")
sourceDirs += file("src/generated/main/grpc")
generatedSourceDirs += file("src/generated/main/java")
generatedSourceDirs += file("src/generated/main/grpc")
}
}

For both projects you can use the repositories.gradle file

repositories {
mavenCentral()
gradlePluginPortal()
}

And the settings.gradle file

pluginManagement {
apply from: 'repositories.gradle', to: pluginManagement
}
rootProject.name = 'location-service' //or sensor-service

If we make a call for any actuator’s route the request we’ll be exported to the opentelemetry collector and we don’t want that (for example, exporting traces from health check and prometheus calls doesn’t seem to be a good practice😉) .

To fix this we need to configure the following class creating the SpanExportingPredicate and ObservationFilter bean.

import io.micrometer.observation.Observation;
import io.micrometer.observation.ObservationFilter;
import io.micrometer.tracing.Tracer;
import io.micrometer.tracing.annotation.*;
import io.micrometer.tracing.exporter.SpanExportingPredicate;
import org.springframework.beans.factory.BeanFactory;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import java.util.Objects;

@Configuration(proxyBeanMethods = false)
public class SpanConfiguration {

@Bean
public NewSpanParser newSpanParser() {
return new DefaultNewSpanParser();
}

@Bean
public MethodInvocationProcessor methodInvocationProcessor(NewSpanParser newSpanParser, Tracer tracer, BeanFactory beanFactory) {
return new ImperativeMethodInvocationProcessor(newSpanParser, tracer, beanFactory::getBean, beanFactory::getBean);
}

@Bean
public SpanAspect spanAspect(MethodInvocationProcessor methodInvocationProcessor) {
return new SpanAspect(methodInvocationProcessor);
}

@Bean
public ObservationFilter urlObservationFilter() {
return context -> {
if (context.getName().startsWith("spring.security.")) {
Observation.Context root = getRoot(context);
if (root.getName().equals("http.server.requests")) {
context.addHighCardinalityKeyValue(Objects.requireNonNull(root.getHighCardinalityKeyValue("http.url")));
}
}
return context;
};
}

private Observation.Context getRoot(Observation.Context context) {
if (context.getParentObservation() == null) {
return context;
} else {
return getRoot((Observation.Context) context.getParentObservation().getContextView());
}
}

@Bean
public SpanExportingPredicate actuatorSpanExportingPredicate() {
return span -> {
if (span.getTags().get("http.url") != null) {
return !span.getTags().get("http.url").startsWith("/actuator");
}
return true;
};
}

}

Now, we need to create the Otel Configuration

import io.opentelemetry.context.propagation.TextMapPropagator;
import io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporter;
import io.opentelemetry.extension.trace.propagation.JaegerPropagator;
import lombok.RequiredArgsConstructor;
import org.springframework.boot.actuate.autoconfigure.tracing.otlp.OtlpProperties;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
@RequiredArgsConstructor
@EnableConfigurationProperties(OtlpProperties.class)
public class OtelConfiguration {

private final OtlpProperties otlpProperties;

@Bean
public TextMapPropagator jaegerPropagator() {
return JaegerPropagator.getInstance();
}

@Bean
public OtlpGrpcSpanExporter otlpExporter() {
return OtlpGrpcSpanExporter.builder()
.setEndpoint(this.otlpProperties.getEndpoint())
.setTimeout(this.otlpProperties.getTimeout())
.build();
}

}

Finally let’s configure the application.yml file

management:
tracing:
enabled: true
sampling:
probability: 1.0
otlp:
tracing:
endpoint: http://localhost:4317
endpoints:
web:
exposure:
include: '*'
metrics:
distribution:
percentiles-histogram:
http:
server:
requests: true

In order to add traces server and client gRPC, we need to configure interceptors as below.

import io.grpc.ClientInterceptor;
import io.grpc.ServerInterceptor;
import io.opentelemetry.api.OpenTelemetry;
import io.opentelemetry.instrumentation.grpc.v1_6.GrpcTelemetry;
import net.devh.boot.grpc.client.interceptor.GrpcGlobalClientInterceptor;
import net.devh.boot.grpc.common.util.InterceptorOrder;
import net.devh.boot.grpc.server.interceptor.GrpcGlobalServerInterceptor;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.annotation.Order;

@Configuration
public class OtelTraceInterceptorConfiguration {

@Bean
@GrpcGlobalServerInterceptor
@Order(InterceptorOrder.ORDER_TRACING_METRICS + 1)
public ServerInterceptor serverInterceptor(OpenTelemetry openTelemetry) {
return GrpcTelemetry.create(openTelemetry).newServerInterceptor();
}

@Bean
@GrpcGlobalClientInterceptor
public ClientInterceptor clientInterceptor(OpenTelemetry openTelemetry) {
return GrpcTelemetry.create(openTelemetry).newClientInterceptor();
}

}

The application development will be explained in part 2.

Free

Distraction-free reading. No ads.

Organize your knowledge with lists and highlights.

Tell your story. Find your audience.

Membership

Read member-only stories

Support writers you read most

Earn money for your writing

Listen to audio narrations

Read offline with the Medium app

Gabriel Martins
Gabriel Martins

Written by Gabriel Martins

Just a brazilian guy who loves technology and wants to share the experiences he has learned

Responses (1)

Write a response

This implementation is great, but tries to send traces on both HTTP and gRPC since we have the default exporter plus another spanExporter defined in this OtelConfiguration. What you need to do on the OtelConfiguration is to override…

1

Recommended from Medium

Lists

See more recommendations