BoxLang AI 3.2.0 is here, and it's a landmark release. We're shipping five major features: image generation, web search, a fluent audio builder API, a centralized agent registry, and deep MCP observability along with a suite of analytics improvements and a critical bug fix. Let's dig in.

Image Generation — aiImage()
You can now generate images directly from BoxLang using any provider that supports text-to-image generation. The aiImage() BIF follows the same fluent, chainable philosophy as the rest of bx-ai then act on the result with expressive method calls.

// Generate and save in one fluent chain
aiImage( "A futuristic cityscape at sunset" )
    .saveToFile( "/images/cityscape.png" )

// Full control with params and provider
response = aiImage(
    "A watercolor painting of a mountain lake",
    { n: 2, size: "1024x1024", quality: "hd" },
    { provider: "openai" }
)

// Embed directly in HTML output
dataURI = response.toDataURI()

The returned AiImageResponse object gives you everything you need: hasImages(), getCount(), getFirstURL(), getFirstBase64(), saveToFile(), saveAllToDirectory(), toDataURI(), getMimeType(), and toStruct().

Supported providers out of the box:

A generateImage@bxai agent tool is auto-registered in the global tool registry at module startup, so your agents can generate images without any manual wiring:

agent = aiAgent( tools: [ "generateImage@bxai" ] )

Image Generation Docs

Web Search — aiWebSearch() & aiWebSearchAsync()
BoxLang AI now ships a unified web search system with provider abstraction and normalized results. Every provider returns the same fields — title, url, snippet, publishedDate, domain, score, thumbnail, language — so you can swap providers without touching your code.

// Synchronous search
results = aiWebSearch( "latest BoxLang AI updates", { provider: "brave", maxResults: 8 } )

// Async — returns a BoxFuture
future = aiWebSearchAsync( "BoxLang release highlights", { provider: "tavily" } )
results = future.get()

Supported providers:

The webSearch@bxai tool is auto-registered globally, so any agent can search the web immediately:

agent = aiAgent(
    name: "ResearchAgent",
    tools: [ "webSearch@bxai" ]
)

response = agent.run( "Find and summarize recent BoxLang AI release highlights" )

Web Search Docs

Fluent Builder API for Audio BIFs
aiSpeak(), aiTranscribe(), and aiTranslate() now support a full fluent builder API. Call any of them with no arguments to get the request object back, then chain your configuration before executing. The traditional positional-argument syntax continues to work exactly as before — the fluent builder is purely additive.

aiSpeak()

// Traditional syntax — still works
audio = aiSpeak( "Hello!", { voice: "nova" }, { provider: "openai" } )

// Fluent builder — expressive and self-documenting
audio = aiSpeak()
    .of( "Hello, world!" )
    .voice( "nova" )
    .provider( "openai" )
    .asMP3()
    .speak()

// Gender shortcuts
audio = aiSpeak()
    .of( "Welcome aboard!" )
    .male()
    .speed( 1.2 )
    .speak()

// Format shortcuts
audio = aiSpeak()
    .of( "System alert." )
    .asWav()
    .outputFile( "/audio/alert.wav" )
    .speak()

Key builder methods: .of(), .voice(), .male() / .female(), .speed(), .instructions(), .outputFile(), .asMP3() / .asWav() / .asFlac() / .asOpus() / .asPCM(), .provider(), .speak().

aiTranscribe()

// From file
text = aiTranscribe()
    .file( "/audio/meeting.mp3" )
    .withWordTimestamps()
    .asVerboseJSON()
    .transcribe()

// From URL
text = aiTranscribe()
    .url( "https://example.com/audio.mp3" )
    .language( "es" )
    .transcribe()

// Translate audio directly to English
english = aiTranscribe()
    .file( "/audio/french.mp3" )
    .translate()

Key builder methods: .file(), .url(), .data(), .language(), .withWordTimestamps(), .withSegmentTimestamps(), .diarize(), .asJSON() / .asText() / .asVerboseJSON() / .asSRT() / .asVTT(), .transcribe(), .translate().

aiTranslate()

english = aiTranslate()
    .file( "/audio/german.mp3" )
    .asText()
    .translate()

Audio Docs

Agent Registry — aiAgentRegistry()
3.2.0 introduces the AIAgentRegistry — a global singleton that gives you centralized discoverability, observability, and lifecycle management for all agents running in your BoxLang application.

// Auto-register at creation time
agent = aiAgent(
    name: "support-agent",
    description: "Customer support agent",
    register: true,
    module: "my-app"
)

// Or register manually
aiAgentRegistry().register( agent, "my-app" )

// Discover what's running
agents = aiAgentRegistry().listAgents()
info   = aiAgentRegistry().getAgentInfo( "support-agent@my-app" )

// Resolve a mixed array of string keys and live instances
resolved = aiAgentRegistry().resolveAgents( [
    "support-agent@my-app",
    anotherAgentInstance
] )

// Clean up
aiAgentRegistry().unregister( "support-agent@my-app" )
aiAgentRegistry().unregisterByModule( "my-app" )

Module Authors: First-Class Agent & Tool Registration
This is a big deal for the BoxLang ecosystem. Developers building BoxLang modules can now ship agents and tools that auto-register themselves globally when the module loads — no manual wiring by the application developer required.

Define your aiAgent() instances with register: true and a module namespace
Define your tools, scan them via aiToolRegistry().scan( new MyTools(), "my-module" ), and they appear globally as toolName@my-module
Application developers can consume your agents and tools by name, from any part of their app, the moment your module is installed
This makes bx-ai a genuine platform for building composable, discoverable AI ecosystems — publish a module to ForgeBox, and your agents and tools show up ready to use.

Two new interception points fire on registry changes: onAIAgentRegistryRegister and onAIAgentRegistryUnregister.

MCP Server Pause/Resume
MCPServer now supports pausing and resuming without tearing down configuration or losing registered tools. Ideal for maintenance windows, graceful degradation, or controlled rollouts.

server = MCPServer( "my-tools", "Provides custom tools" )
    .registerTool( myTool )

server.pause()

if ( server.isPaused() ) {
    println( "Server is paused — rejecting all non-ping requests" )
}

server.resume()

pause() — fires onMCPServerPause; all non-ping requests receive error code -32005
resume() — fires onMCPServerResume; normal handling restored
getSummary() now includes a paused boolean
MCP Server & Client Observability
Server Analytics
MCP server monitoring gets a major overhaul in 3.2.0:

Thread-safe counters using named locks across all stat operations
Security failure tracking — auth failures, API key rejections, body-size violations all get dedicated counters
Per-tool error tracking — byTool[name].errors with errors.byTool roll-up
Active concurrent request counter — activeRequests increments and decrements in real time
Requests-per-minute rate — exposed in getSummary()
X-Request-ID correlation — request IDs echoed in response headers and event payloads
Paused-request stats — rejected requests tracked when server is paused
onMCPError now fires for METHOD_NOT_FOUND
Client Stats — MCPClient
MCPClient gains full internal usage and performance tracking:

client = MCP( "http://localhost:3000" )

tools  = client.listTools()
result = client.callTool( "search", { query: "BoxLang" } )

// Inspect what's happening
stats   = client.getStats()   // per-operation, per-tool, per-URI breakdowns
summary = client.getSummary() // totalCalls, successRate, avgResponseTime

// Reset when needed
client.resetStats()

Three new interception points cover the full client lifecycle: onMCPClientRequest, onMCPClientResponse, onMCPClientError.

Type-Aware Tool Argument Support
Tool schemas in bx-ai are now generated directly from callable parameter metadata, so LLMs finally receive accurate JSON Schema types for every argument instead of a flat bag of strings. ClosureTool.getArgumentsSchema() maps BoxLang types naturally — numeric, integer, float, and double become "number", boolean becomes "boolean", array becomes "array" with "items": {}, and struct becomes "object" — meaning LLMs can send native JSON values for non-string arguments and tools behave exactly as their signatures declare. On the output side, BaseTool.invoke() continues to serialize results consistently for provider compatibility, converting simple values via toString() and complex values via JSON serialization, keeping the tool contract clean in both directions.

// Tool with numeric and boolean arguments
// LLM sends { "quantity": 3, "applyDiscount": true } — no casting needed
calculateTotal = aiTool(
    name: "calculateTotal",
    description: "Calculate order total with optional discount",
    tool: ( numeric price, numeric quantity, boolean applyDiscount = false ) -> {
        total = price * quantity
        if ( applyDiscount ) total *= 0.9
        return { summary: "Order total calculated", total: total }
    }
)

// Tool with an array argument
// LLM sends { "tags": ["boxlang", "ai", "tools"] } — native array
tagContent = aiTool(
    name: "tagContent",
    description: "Apply a list of tags to a content item",
    tool: ( string contentId, array tags ) -> {
        // tags arrives as a real BoxLang array
        return {
            summary : "Tags applied to #contentId#",
            applied : tags.len(),
            tags    : tags
        }
    }
)

// Tool with a struct argument
// LLM sends { "filter": { "status": "active", "minAge": 18 } } — native struct
queryUsers = aiTool(
    name: "queryUsers",
    description: "Query users by filter criteria",
    tool: ( struct filter, numeric limit = 10 ) -> {
        results = userService.query( filter, limit )
        return {
            summary : "Found #results.len()# users",
            count   : results.len(),
            data    : results
        }
    }
)

agent = aiAgent(
    tools: [ calculateTotal, tagContent, queryUsers ]
)

Bug Fix — ClosureTool.doInvoke() JSON Struct Handling
MCP clients that send JSON fields as real objects or arrays (rather than pre-stringified JSON) no longer cause "Can't cast Struct to a string" errors. doInvoke() now inspects declared parameters and calls jsonSerialize() on any non-simple value whose declared type is string. Silent, automatic, no code changes required.

Module Configuration
New image Settings Block

{
  "modules": {
    "bxai": {
      "settings": {
        "image": {
          "defaultProvider": "openai",
          "defaultApiKey": "",
          "defaultModel": "gpt-image-1",
          "defaultSize": "1024x1024",
          "defaultQuality": "standard",
          "defaultStyle": "vivid",
          "defaultInstructions": ""
        }
      }
    }
  }
}

New Interception Points
3.2.0 brings bx-ai to 50 total interception points, adding 10 new events:

Upgrade Now

# CommandBox
box install bx-ai

# OS
install-bx-module bx-ai

Full Docs: ai.ortusbooks.com Community: community.ortussolutions.com GitHub: github.com/ortus-boxlang/bx-ai

BoxLang AI 3.2.0 is a platform release: image generation, web search, fluent audio, a global agent & tool registry, and deep observability all land together. We can't wait to see what you build.

The post BoxLang AI 3.2.0 — Image Generation, Web Search, Fluent Audio, Agent Registry & MCP Observability appeared first on foojay.

article: Exclusive: Why are Chinese AI models dominating open-source as Western labs step back?
authors: Dashveenjit Kaur, AI News
date: 2026-02-09
desc.: A shift in what AI models are being used and where the models are being produced.
category: opinion

article: Machines of Loving Grace
authors: Dario Amodei
date: 2024-10-01
desc.: Although the article is older, it remains relevant for any author aiming to sketch a future in which everything with AI goes right. In light of recent developments, which appear to follow a sigmoid curve rather than exponential growth (marked by stagnation, with current models reaching a point where another breakthrough is required), the trajectory looks more measured than initially anticipated. Although the author discusses multiple risks (grandiosity, market forces, propaganda, sci-fi-like expectations, etc.), he also highlights the bright sides and explores areas where current AI may prove genuinely helpful. The question remains whether the current state of affairs can truly guarantee progress, rather than causing damage through non-deterministic outcomes (education, industry, human creativity etc.).
category: opinion

article: The Urgency of Interpretability
authors: Dario Amodai
date: 2025-04-01
desc.: The author describes lessons learned from current AI development and adds multiple valuable thoughts and facts to consider when interacting with AI models. The main point is that progress in the underlying technology is inexorable, driven by forces too powerful to stop, but what matters is the way in which it unfolds. Accepting that the current evolution of LLM-based AI cannot be halted, the author expresses hope that it may still be guided (this fact affect not only entire industry but also human kind thoughs and perception of reality), much like a bus controlled by a steering wheel, and warns of the dangers of ignorance, illustrating this through several concrete examples.
category: opinion

article: From Delegates to Trustees: How Optimizing for Long-Term Interests Shapes Bias and Alignment in LLM
authors: Suyash Fulay, Jocelyn Zhu, Michiel Bakker (MIT)
date: 2025-10-14
desc.: The article addresses the question of 'behavioral cloning', specifically, how accurately LLMs reproduce individuals' expressed preferences. Large language models have demonstrated promising accuracy in predicting survey responses and policy preferences, which has fueled growing interest in their potential to represent human interests across various domains. Drawing on theories of political representation, the article highlights an underexplored design trade-off: whether AI systems should act as delegates, mirroring expressed preferences, or as trustees, acting in users' broader interests. Models may align well with users' short-term preferences while failing to account for their long-term interests. Studies further indicate greater bias in topics where consensus is lacking.
category: research

article: DARE-bench: Evaluating Modeling and Instruction Fidelity of LLMs in Data Science
authors: Fan Shu, Yite Wang, Ruofan Wu, Boyi Liu, Zhewei Yao, Yuxiong He, Feng Yan
date: 2026-02-27
desc.: The article addresses the challenge posed by fast-growing demand for Large Language Models (LLMs) to tackle complex, multi-step data science tasks, which has created an urgent need for accurate benchmarking. Two major gaps are identified in existing benchmarks: (i) the lack of standardized, process-aware evaluation that captures instruction adherence and process fidelity, and (ii) the scarcity of accurately labeled training data. While highlighting that even capable models (Anthropic, OpenAI, etc.) may struggle in performance, the article introduces the DARE-bench benchmark alongside supervised fine-tuning as approaches that may improve outcomes in specific applications. Although the results appear promising, they retain considerable potential for further improvement, as accuracy is not yet guaranteed.
category: research

article: Do LLMs Benefit From Their Own Words?
authors: Jenny Y. Huang, Leshem Choshen, Ramon Astudillo, Tamara Broderick, Jacob Andreas (MIT, IBM Research)
date: 2026-02-27
desc.: The article aims to answer the question of whether preserving past assistant responses is more beneficial than harmful. The study uses in-the-wild, multi-turn conversations and compares standard (full-context) prompting with a user-turn-only prompting approach that omits all previous assistant responses, evaluated across three open reasoning models and one state-of-the-art model. Surprisingly, omitting past assistant responses does not negatively affect response quality in a large fraction of turns and may also reduce token length. The article concludes with a discussion of findings and directions for future research.
category: research

article: SafeGen-LLM: Enhancing Safety Generalization in Task Planning for Robotic Systems
authors: Jialiang Fan, Weizhe Xu, Mengyu Liu, Oleg Sokolsky, Insup Lee, Fangxin Kong
date: 2026-02-27
desc.: Safety-critical task planning in robotic systems remains a significant challenge: classical planners suffer from poor scalability, reinforcement learning (RL)-based methods generalize poorly, and base large language models (LLMs) cannot guarantee safety. To address this gap, the article proposes SafeGen-LLM, a safety-generalizable large language model framework. As part of this contribution, a multi-domain Planning Domain Definition Language 3 (PDDL3) benchmark with explicit safety constraints is introduced, along with Supervised Fine-Tuning (SFT) on those constraints. Although the results appear optimistic, with minimal safety violations observed across tested domains, the approach still requires further research in more complex robotic settings.
category: research

article: LemmaBench: A Live, Research-Level Benchmark to Evaluate LLM Capabilities in Mathematics
authors: Antoine Peyronnet, Fabian Gloeckle, Amaury Hayat
date: 2026-02-27
desc.: Existing benchmarks largely rely on static, hand-curated sets of contest or textbook-style problems as proxies for mathematical research. The article introduces a novel approach leveraging state-of-the-art models (GPT-5, Gemini 2.5, Gemini 3, Claude Opus 4.5, and DeepSeek-R) by extracting lemmas from arXiv and updating them dynamically. This results in a benchmark that can be refreshed regularly with new problems drawn directly from current mathematical research, while previous instances can be used for training without compromising future evaluations. This approach achieves 10–15% accuracy in theorem proving and opens a new frontier for future research. Although the process may appear fully automated, a human in the loop, such as the article's author or reviewer, remains critically necessary to produce high-quality inputs and to effectively use LLM models.The results also indicate that it is considerably easier for a model to validate an existing proof than to produce one.
category: research

article: Task Complexity Matters: An Empirical Study of Reasoning in LLMs for Sentiment Analysis
authors: Donghao Huang, Zhaoxia Wang
date: 2026-02-27
desc.: It is a well-established narrative that reasoning in large language models (LLMs) universally improves performance across language tasks. This article aims to test that claim through a comprehensive evaluation of 504 configurations across seven models, considering different reasoning architectures such as adaptive, conditional, and reinforcement-based approaches. The findings reveal that the effectiveness of reasoning is strongly task-dependent and degrades for simpler tasks. The article provides quantitative findings alongside error analysis and outlines directions for future research.
category: research

article: Benchmarking LLM Summaries of Multimodal Clinical Time Series for Remote Monitoring
authors: Aditya Shukla, Yining Yuan, Ben Tamo, Yifei Wang, Micky Nnamdi and others
date: 2026-03-02
desc.: Large language models (LLMs) can generate fluent clinical summaries of remote therapeutic monitoring time series, however, the impact of information bias on clinically significant events, such as sustained abnormalities, remains poorly understood. The article presents the Technology-Integrated Health Management (TIHM) framework to address these questions, introducing a protocol that measures abnormality recall, duration recall, and measurement coverage, while utilizing GPT-4o-mini as a proxy evaluator. Traditional models frequently exhibit near-zero abnormality recall, whereas the vision-based approach achieves the strongest event alignment, with 45.7% abnormality recall and 100% duration recall. These results underscore the need for event-aware evaluation methods in future research to ensure reliable clinical time-series summarization.
category: research

article: Full interview: Anthropic CEO responds to Trump order, Pentagon clash
authors: CBS News
date: 2026-02-28
desc.: Anthropic CEO Dario Amodei sat down with CBS News for an exclusive interview, hours after Defense Secretary Pete Hegseth declared the company a supply chain risk to national security, which restricts military contractors from doing business with the AI giant. Amodei called the move "retaliatory and punitive," and he said Anthropic sought to draw "red lines" in the government's use of its technology because "we believe that crossing those lines is contrary to American values, and we wanted to stand up for American values.". Response of the OpenAI striking a deal with Pentagon causes many questions.
category: youtube

article: Scary Agent Skills: Hidden Unicode Instructions in Skills ...And How To Catch Them
authors: Embrace The Red
date: 2026-02-11
desc.: Skills introduce common threats such as prompt injection, supply chain attacks, remote code execution (RCE), and data exfiltration, among others. This post discusses the fundamentals, highlights the most straightforward prompt injection vector, and demonstrates how a real Skill from OpenAI can be back-doored using invisible Unicode Tag code-points, a technique that certain models, including Gemini, Claude, and Grok, are known to interpret as instructions. From a security perspective, Skills present serious concerns, as they represent a typical supply chain risk with limited governance or security controls. The author identified that some Skills instruct the AI to embed API tokens directly in curl requests and similar constructs , a poor design practice. This means that credentials are passed through the LLM, making them susceptible to leakage and leaving them vulnerable to being overwritten by an attacker via indirect prompt injection.
category: tutorial

The post JC-AI Newsletter #14 appeared first on foojay.

In the era of vibe coding—where large amounts of code are introduced or refactored in short bursts, often with the help of LLMs—you need immediate feedback on how new logic actually executes. Not comprehensive analysis. Not nanosecond-precise timing. Just a quick confirmation that your loops aren't spinning 10,000x more than they should.

However, traditional profilers can feel like overkill for quick validation. In addition, they present results at method/stack granularity and require context-switching to interpret. They also introduce overhead, ranging from negligible (e.g., JFR/sampling) to noticeable (call tracing/instrumentation). As a result, they are less convenient as always-on feedback during rapid iteration.

jvm-hotpath is a lightweight Java agent built for this workflow. It surfaces per-line execution counts directly in your source code, showing you exactly which lines run and how often—while your application runs.

What Makes This Different

Zero timing overhead. Just counts, no nanosecond measurements.

Counts every execution. No sampling, no missed fast methods.

LLM-friendly output. JSON reports you can pipe to an LLM for analysis.

Live updates. JSONP polling lets you watch counts update live. No server is needed.

Modern Java. Tested in CI on Java 11, 17, 21, 23, and 24. It also works with Spring Boot and Micronaut.

The Gap in Java Tooling

The Original Problem

The immediate pain is simple: code arrives faster than you can build a mental model of it. Years ago, I faced the same core problem in an inherited system. So I hacked Cobertura—a coverage tool—to use it as a runtime analysis tool. By instrumenting the app and exercising specific behaviors, I could observe execution counts after the fact. As a result, I got a runtime-shaped mental map of the codebase—and an entry point for making changes with confidence.

After all, static analysis tells you what could execute. Tests tell you what should execute. What I needed was to see what does execute under real workloads.

Why Existing Tools Don't Fit

Cobertura's last release was in 2015, so it doesn't fit modern Java toolchains. Since then, no widely adopted, actively maintained tool has focused on live per-line execution frequency.

Coverage tools (e.g., JaCoCo) track whether code executed, not how many times. Profilers show where CPU time goes. Neither one shows you execution frequency under real conditions.

How I Ended Up Building This

Modern Java tooling has moved in different directions, but the idea stuck with me. So I evaluated what was available. For instance, OpenClover's "full support" line is Java 17, with newer versions listed as experimental. Similarly, JCov exists as an OpenJDK CodeTools project, but setup is old-school. In short, there's no simple "pull a jar from Maven Central and go" path. IntelliJ's built-in coverage is excellent for coverage, and it stores run data as IDE coverage suites (e.g., .ic). However, it's still an IDE-centric workflow. In short, it's not something you can reuse in CI artifacts or share as a standalone live report.

After an hour of dead ends, Claude cut to the chase:

"Do you want me to help you build JCov from source, or would you rather I create a simple custom execution counter for you?"

Ultimately, that question decided the direction.

A Real-World Bug

How the Bug Appeared

This tool was born during a high-velocity vibe coding session. Specifically, I was refactoring a core processing engine. Standard profilers missed this bug. The system didn't feel slow yet:

The Bug: A .filter(r -> r.isDuplicate()) call was executing 19 million times in 15 seconds.
The Problem: Each call was ~50 nanoseconds—easy for sampling profilers to under-sample.
The Impact: O(N²) instead of O(1) was hiding in plain sight.

Why It Was Hard to Spot

In other words, the filter was sitting inside a loop instead of being evaluated once. It's a classic mistake. Yet it's invisible to traditional tools. Instead, I wanted immediate runtime visibility into what was actually running.

Execution counts made it obvious. For example, seeing "19,147,293 executions" next to a single line removed all ambiguity. No timing data was required, and no interpretation was needed.

The Key Insight: Frequency ≠ Resource Consumption

Java profilers focus on resource consumption: CPU time, memory allocation, thread contention. jvm-hotpath, by contrast, shows how many times code runs (frequency).

In modern Java, this distinction matters. For instance, JIT compilation makes individual calls fast. So the bottleneck is often algorithmic—O(N) vs O(1). Also, logic errors can create millions of unnecessary calls. Furthermore, sampling profilers are statistical. Moreover, very short but frequent work is easy to under-sample.

It's a "Logic X-Ray," not a "Resource Monitor."

How It Works

Instrumentation

jvm-hotpath is a Java agent that instruments bytecode at class-load time using ASM. Specifically, it inserts a counter before each executable line. Indeed, there's no sampling, no timing—just frequency.

As a result, the overhead is low enough for normal development runs.

The Report

The collected data is written to an interactive HTML report that refreshes while your app runs. Specifically, it shows syntax-highlighted source code with execution counts next to each line. In addition, a global heatmap makes hot paths stand out visually.

JSONP-powered polling lets you open the report directly from disk (file://) and watch it update live. No server is needed.

Notably, the narrow focus is intentional. There are no flame graphs, no dashboards, no post-hoc traces—just line-level execution frequency mapped onto source code.

Machine-Readable Output

The agent also writes execution-report.json. Therefore, it gives you a machine-readable artifact you can feed into CI steps or LLM-based tools.

See it in action:

https://github.com/user-attachments/assets/cc89451b-a41f-491e-a1f6-8e87328979c0

Getting Started

Maven Plugin (Recommended)

Add the plugin to your pom.xml:

<plugin>
    <groupId>io.github.sfkamath</groupId>
    <artifactId>jvm-hotpath-maven-plugin</artifactId>
    <version>0.2.4</version>
    <executions>
        <execution>
            <goals>
                <goal>prepare-agent</goal>
            </goals>
        </execution>
    </executions>
    <configuration>
        <!-- Auto-flush report every 5 seconds -->
        <flushInterval>5</flushInterval>
    </configuration>
</plugin>

Then run your application with the agent active:

mvn -Pinstrument jvm-hotpath:prepare-agent exec:exec

For exec:exec, you need a main class. Pass -Dexec.mainClass=... or configure exec.mainClass in pom.xml.

The report is generated at target/site/jvm-hotpath/execution-report.html.

For multi-module projects or generated code (OpenAPI/MapStruct), the plugin can merge multiple source roots into one report. Also, you can pass dependency source archives directly via sourcepath.

Manual Agent Usage

If you prefer direct control, run:

java -javaagent:jvm-hotpath-agent.jar=packages=com.example,sourcepath=src/main/java,flushInterval=5 -jar your-app.jar

Key parameters: packages sets which packages to instrument. sourcepath points to source roots or archives (.jar, .zip). flushInterval controls seconds between report refreshes (0 = no auto-flush). verbose prints instrumentation details with clickable file URLs.

Standalone Report Regeneration

If you already have execution-report.json from CI, you can regenerate the HTML without rerunning the application:

java -jar jvm-hotpath-agent.jar --data=target/site/jvm-hotpath/execution-report.json --output=target/site/jvm-hotpath/new-report.html

What This Is Not

It is not a coverage percentage tool—use JaCoCo for that. Nor is it a CPU timing profiler—use JFR or async-profiler instead. Finally, it is not a 24/7 production monitoring system.

Beyond Performance: Dead Code and Cognitive Load

Execution counts make it easy to spot dead code and rarely used branches. Moreover, they surface features that exist largely for historical reasons. Furthermore, they reduce cognitive load. When you know which parts actually run, it becomes much easier to reason about changes, refactor with confidence, or decide what not to think about yet.

Indeed, for anyone working quickly with AI-assisted tools, that kind of clarity is invaluable.

A Note on How This Was Built

The first prototype came out of AI-assisted vibe coding, primarily with Claude. Subsequently, I iterated with a mix of manual work and help from Codex and Gemini. I also validated everything against real JVM workloads.

Overall, the tools accelerated exploration. Even so, the motivation and direction came from hands-on use in real codebases.

Where This Is Going

There are obvious next steps—Gradle improvements, better exclusion controls, broader framework testing. For now, though, I'm deliberately keeping the scope small. Indeed, this is my first open-source release.

The real question is simpler: does this help you understand your codebase faster and with more confidence?

Project: github.com/sfkamath/jvm-hotpath
Documentation: Full README
Motivation: Deep dive into the why

The post Runtime Code Analysis in the Age of Vibe Coding appeared first on foojay.

Profiling a Java Application Running Inside an Unikernel with JProfiler

Unikernels are often associated with minimalism and tight resource control.

But can we profile a Java application running inside a unikernel using a standard JVM profiler?

The answer is yes.

In this guide, we will walk step by step through profiling a Quarkus
application running inside a Nanos unikernel using JProfiler and
IBM Semeru JRE 25 (OpenJ9).

No special hacks. Just standard JVM tooling.

Prerequisites

• Ops (https://ops.city/)
• JProfiler (https://www.ej-technologies.com/jprofiler)
• A Quarkus application packaged as a runnable JAR

1. Project Structure

We start from a clean Proof of Concept directory:

quarkusProfiling$ ls
config.json  jprofiler15  myQuarkusApp  tmp

Directory breakdown:

• config.json → Ops/Nanos configuration file
• jprofiler15/ → extracted JProfiler distribution
• myQuarkusApp/ → packaged Quarkus runnable JAR
• tmp/ → writable directory inside the unikernel

2. Ops Nanos Configuration

Below is the complete config.json used for this setup:

{
    "Version": "1.0.0",
    "Dirs":["myQuarkusApp","jprofiler15","tmp"], 
    "Args": [
        "-agentpath:/jprofiler15/bin/linux-x64/libjprofilerti.so=port=8849,address=0.0.0.0",
        "-jar",
        "myQuarkusApp/quarkus-faces-runner.jar"
    ],
    "RunConfig": {
        "Ports": ["8080","8849"]
    },
    "BaseVolumeSz": "300m"  
}

Explanation

Dirs

"Dirs":["myQuarkusApp","jprofiler15","tmp"]

These directories are embedded into the unikernel filesystem:

• The application
• The JProfiler native agent
• A writable tmp directory

Args

"-agentpath:/jprofiler15/bin/linux-x64/libjprofilerti.so=port=8849,address=0.0.0.0"

This is the key configuration.

We explicitly load the JProfiler native JVMTI agent at JVM startup.

Important parameters:

• port=8849 → port used by the profiler
• address=0.0.0.0 → listen on all interfaces

Then we launch the application:

"-jar", "myQuarkusApp/quarkus-faces-runner.jar"

RunConfig

"Ports": ["8080","8849"]

We expose:

• 8080 → Quarkus HTTP endpoint
• 8849 → JProfiler agent port

BaseVolumeSz

"BaseVolumeSz": "300m"

Allocates 300 MB for the unikernel filesystem.

3. Starting the Unikernel

From the project root directory, run:

ops pkg load AngeloRubens/SemeruJREx64Linux:25.0.1 --nightly -c config.json

What this command does

• Loads IBM Semeru JRE 25 (OpenJ9)
• Applies our local config.json
• Embeds the application and profiler agent
• Boots the unikernel locally

There is no separate container or VM runtime.\
The JVM runs directly inside the unikernel.

4. Boot Log Analysis

When the unikernel starts, you will see:

running local instance
booting /home/lop/.ops/images/java ...
[0.227374] en1: assigned 10.0.2.15

The unikernel has booted and received an IP address.

Then the JVM starts and the profiler agent initializes:

JProfiler> Protocol version 68
JProfiler> OpenJ9 JVMTI detected.
JProfiler> Java 25 detected.
JProfiler> 64-bit library
JProfiler> Listening on port: 8849.

This confirms:

• OpenJ9 JVM detected
• Java 25 runtime
• JVMTI available
• Native agent loaded successfully

Instrumentation Phase

JProfiler> Enabling native methods instrumentation.
JProfiler> Can retransform classes.
JProfiler> Native library initialized
JProfiler> VM initialized

This means:

• Class retransformation is supported
• Native methods can be instrumented
• The agent is fully integrated

Final State

JProfiler> Retransforming 11 base class files.
JProfiler> Base classes instrumented.
JProfiler> Waiting for a connection from the JProfiler GUI ...
JProfiler> Using address 0.0.0.0.

At this point:

• The JProfiler agent is listening on port 8849
• The JVM is waiting for a GUI connection

The profiling infrastructure inside the unikernel is fully operational.

5. Connecting JProfiler GUI

Open JProfiler.

1. Click New Session
2. Select Attach to remote JVM
3. Enter:
   • Host: 127.0.0.1
   • Port: 8849
4. Connect

The GUI will attach to remote JVM running inside the unikernel(or connect to remote jvm running in you cloud provider or onprem).

From here you can:

• Inspect CPU hotspots
• Analyze memory allocations
• Capture heap snapshots
• Monitor threads
• View GC activity

Exactly as you would with a standard JVM.

SO Info(Nanos name):

CPU Hot Spots

CPU Call Tree

Live Objects

Why This Works

Unikernels do not remove JVM capabilities.

JProfiler uses:

• JVMTI
• Native agent loading via -agentpath
• Standard TCP communication

As long as:

• The agent library is available
• The port is exposed
• The JVM supports JVMTI

Profiling works.

The environment (VM, container, unikernel) does not change the JVM instrumentation model.

Conclusion

Profiling a Java application inside a unikernel is straightforward.

By:

• Embedding the profiler native agent
• Exposing the profiler port
• Using a standard JVM (IBM Semeru OpenJ9)

You can use JProfiler --- or any JVMTI-based tool --- without
limitations.

Unikernels do not prevent observability.\
They simply run your JVM in a different execution model.

And from the JVM perspective, everything works exactly as expected.

If you want, yoou can see the sample app of this post, running into nanos Unikernel on Arm64 Ampere Oracle OCI Cloud http://80.225.94.232:8080/index.xhtml

The post Unikernel: Profiling and Troubleshooting JVM on Nanos Unikernel appeared first on foojay.

While significant activity characterizes the AI field, many unresolved research, design, and implementation challenges continue to impact progress. Future advancement depends heavily on understanding the nature of these challenges to approach probabilistic problems from the appropriate directions. This JC-AI newsletter features insightful interviews with key figures in the field, enabling readers to ask the right questions and compare visions of an 'uncertain future' against current capabilities to maintain a grounded perspective.

article: Deep Researcher with Sequential Plan Reflection and Candidates Crossover (Deep Researcher Reflect Evolve)
authors: Saurav Prateek
date: 2026-01-28
desc.: This paper introduces Deep Researcher, a novel architecture that shifts the paradigm from latency-optimized parallel scaling to an accuracy-driven sequential refinement model. Within the development of Deep Research Agents (DRAs), two primary paradigms are considered, Parallel Scaling and Sequential Refinement. The Deep Researcher agent achieved an overall score of 46.21 on the Research Bench, demonstrating superior performance compared to existing agents, including Claude Researcher, Nvidia AIQ Research Assistant, Perplexity Research, Kimi Researcher, and Grok Deep Search. While these improvements are good, the field requires further research to address remaining challenges.
category: research

article: Manipulation in Prediction Markets: An Agent-based Modeling Experiment
authors: Bridget Smart, Ebba Mark, Anne Bastian, Josefina Waugh (University of Oxford)
date: 2026-01-28
desc.: The paper investigates the utilization of agentic systems in the economic field and their impact on prediction. First, the paper evaluates an agent-based model of a prediction market in which bettors with heterogeneous expertise, noisy private information, variable learning rates, and budgets observe the evolution of public opinion on a binary election outcome to inform their betting strategies in the market. The agentic system exhibits stability across experiments. The second area relates to experiments on how "whale" agents, a highly resourced minority with biased information, may distort market prices and for how long. The paper discusses interesting simulation results on how biased information may change the market from a long-term perspective.
category: research

article: Beyond Accuracy: A Cognitive Load Framework for Mapping the Capability Boundaries of Tool-use Agents
authors: Qihao Wang, Yue Hu, Mingzhe Lu, Jiayue Wu, Yanbing Liu, Yuanmin Tang
date: 2026-01-28
desc.: While LLMs' ability to use external tools enables powerful real-world applications, current benchmarks focus on final accuracy rather than revealing the cognitive bottlenecks that limit their true capabilities. This paper presents a framework based on Cognitive Load Theory that aims to decompose tasks into two components: Intrinsic Load and Extraneous Load. The paper discusses performance inconsistencies as cognitive load increases, and demonstrates how the proposed framework enables the identification of capability boundaries in the examined examples.
category: research

article: Build a Prompt Learning Loop - SallyAnn DeLucia & Fuad Ali, Arize
authors: AI Engineer, Sally Ann Delucia, Fuad Alli (Arize)
date: 2026-01-06
desc.: This talk aims to provide ideas on how it is possible to improve LLM responses by using feedback loops. It's important to view this talk through the lens of current research results regarding the LLM hallucination phenomenon and other factors. The main reason to keep current research results in mind is to avoid ending up in an infinite loop of failure/error.
category: youtube

article: Stanford CS230 | Autumn 2025 | Lecture 8: Agents, Prompts, and RAG
authors: Stanford Online
date: 2025-11-11
desc.: For more information about Stanford’s Artificial Intelligence professional and graduate programs
category: youtube, tutorial

article: Developer Experience in the Age of AI Coding Agents – Max Kanat-Alexander, Capital One
authors: AiEngineer, Max Kanat-Alexander
date: 2025-12-23
desc.: It feels like every two weeks, the world of software engineering is being turned on its head. Are there any principles we can rely on that will continue to hold true, and that can help us prepare for the future, no matter what happens? Max uses research, data, and his 20+ years working in enterprise Developer Experience teams to talk through what we can do now that will prepare us for an agentic future, no matter what that future holds.
category: youtube, opinion

article: Token-Guard: Towards Token-Level Hallucination Control via Self-Checking Decoding
authors: Yifan Zhu, Huiqiang Rong, Haoran Luo
date: 2026-01-29
desc.: Hallucination is a recognized phenomenon in the LLM field that impacts applications such as Retrieval-Augmented Generation (RAG) and Reward Modeling (RM). This paper introduces Token-Guard, a self-checking mechanism designed to identify and control hallucinations at the token level. The experiments demonstrate improvements.
category: research

article: Reward Models Inherit Value Biases from Pretraining
authors: Brian Christian, Jessica A. F. Thompson, Elle Michelle Yang, Vincent Adam, Hannah Rose Kirk and others (University of Oxford, University Pompeu Farba)
date: 2026-01-28
desc.: Despite their importance in LLM alignment, reward models (RMs) remain under-researched. This paper provides evidence that RMs inherit biases from their base models, suggesting that the choice of an open-source model is a reflection of values as much as performance. The paper discusses limitations of experiments and offers avenues for future research.
category: research

article: Professor Geoffrey Hinton - AI and Our Future
authors: City of Hobart, Geoffrey Hinton
date: 2026-01-08
desc.: Professor Geoffrey Hinton, known as the "Godfather of AI", will discuss artificial intelligence - how it works, the risks it poses to our society, and how we might coexist with super-intelligent AI. Ideal for business leaders, creatives, researchers, educators, students and anyone curious about the future of intelligence and society.
category: opinion

article: Your MCP Server is Bad (and you should feel bad) - Jeremiah Lowin, Prefect
authors: AI Engineer, Jeremiah Lowin
date: 2026-01-12
desc.: Too many MCP servers are simply glorified REST wrappers, regurgitating APIs that were designed for SDKs rather than agents. This leads to confused LLMs, wasted tokens, and demonstrably poor performance. If you have ever pointed an MCP generator at an OpenAPI spec and called it a day, this talk is your wake-up call.
category: youtube

article: Frontier Models & AI | Sam Altman, CEO & Co-Founder, OpenAI
authors: Cisco
date: 2026-02-04
desc.: Although Sam Altman, CEO and Co-Founder of @OpenAI, explores ideas about future possibilities and potential developments, he is asked during the interview to align his vision with the current state of research and existing technological capabilities. The interview, however, does not present clear data demonstrating how Codex outperforms alternatives or what 'better' specifically means in this context. The responses to questions may appear to be non-deterministic in nature. The interview relies heavily on thoughts about an "undefined future" that would require a deterministically defined foundation. It is interesting how the interview examined frontier AI models and their implications for economies, institutions, and global systems.
category: opinion

article: How to build secure and scalable remote MCP servers
authors: Den Delimarsky (Microsoft)
date: 2025-07-25
desc.: The tutorial provides insights into how to build a reliable Model Context Protocol (MCP) server, enabling AI agents to connect to external tools. It covers several crucial areas and provides valuable resources and ideas for tackling the challenge.
category: tutorial

The post JC-AI Newsletter #13 appeared first on foojay.

This article explores how Quarkus can help organizations reduce costs, streamline development, and modernize their Java applications for today’s cloud-native environments. It outlines the real-world benefits of adopting Quarkus and highlights how its core features address the performance and scalability challenges commonly associated with traditional Java frameworks.

Quarkus is already being adopted across industries. One example is Orange, a global telecom provider that selected Quarkus to support its 5G API initiative, and benefited from fast startup times, a lightweight footprint, and seamless integration with Kubernetes.

After evaluating multiple frameworks, Orange chose Quarkus as the optimal solution for exposing 5G APIs, thanks to its fast startup, lightweight footprint, modularity, and seamless Kubernetes deployment. Quarkus successfully deployed 10 APIs across 4G/5G network cores, with smooth upgrades and optimized resource usage. This solidified Quarkus as a key technology for telecom innovation.

For a collection of user stories from the community, see the Quarkus user stories blog series. These stories highlight how different teams and organizations are using Quarkus in the real world.

What does Quarkus have to offer?

Developer joy with live coding and dev mode

Quarkus streamlines the traditional write-compile-deploy-refresh cycle by offering live coding support out of the box. As developers make changes, Quarkus automatically detects, recompiles, and redeploys the application, which eliminates the need for manual restarts.

While similar functionality has existed through third-party tools, Quarkus integrates it natively and without licensing overhead. This significantly boosts productivity and enhances the developer experience.

Cost efficiency and performance

By optimizing for low memory usage and fast startup times, Quarkus enables higher-density deployments and rapid scaling.

For comparable workloads, Quarkus typically consumes fewer resources such as CPU and memory, which can lead to significant cost savings in cloud environments.

However, organizations considering the switch should always measure and evaluate their specific workloads to validate these benefits in practice.

Reactive at its core

At its core, Quarkus is built on Eclipse Vert.x, a high-performance reactive toolkit. Still, it allows developers to work primarily in an imperative style while leveraging the performance benefits of its reactive underpinnings.

This hybrid approach allows developers to squeeze out even more efficiency from traditional imperative programming while offering the flexibility to adopt reactive patterns where they make sense. Unlike traditional reactive-only frameworks, Quarkus enables developers to combine both imperative and reactive styles in a single application.

This is particularly beneficial for systems requiring high throughput and low latency, ensuring that applications remain robust under heavy load.

Quarkus’s reactive model makes it ideal for event-driven architectures and microservices.

A basic example of reactive messaging in Quarkus:

@ApplicationScoped
public class PriceConverter {

    @Incoming("prices")
    @Outgoing("converted-prices")
    public double convert(double priceInEuro) {
        return priceInEuro * 1.1;
    }
}

In this example, prices are received from one channel (prices), converted, and sent to another channel (converted-prices). This pattern supports high-throughput, event-driven processing with clean and efficient logic.

An example of a reactive HTTP endpoint using reactive routes in Quarkus:

@ApplicationScoped
public class GreetingRoute {

    @Route(path = "/hello", methods = HttpMethod.GET)
    public Uni hello() {
        return Uni.createFrom().item("Hello from reactive route!");
    }
}

This route handles HTTP GET requests reactively using Uni from Mutiny, making it easy to build non-blocking, low-latency APIs.

Which of your current development pains could Quarkus solve?

One often-overlooked benefit of Quarkus is how it improves onboarding and standardization across teams.

With built-in conventions, automatic service provisioning, and curated extension defaults, Quarkus helps developers get up to speed quickly and encourages consistent patterns across projects.

Dev Services

Quarkus Dev Services reduce friction during development and testing by automatically provisioning required services such as databases, message brokers, or identity providers.

For example, if your application includes PostgreSQL, Kafka, or Keycloak extensions, Quarkus can spin up the necessary containers without any manual setup. This allows you to focus on coding instead of configuring infrastructure, accelerating your local development workflow.

Vast extension ecosystem

Quarkus offers a rich extension ecosystem that simplifies integration with essential technologies such as databases, messaging systems, authentication providers, and cloud services.

In addition to official extensions, the Quarkiverse community provides a growing collection of open-source extensions maintained by contributors across the ecosystem. This broadens the range of supported technologies and enables developers to benefit from shared solutions and community expertise.

Popular extensions include:

• quarkus-hibernate-orm and quarkus-jdbc-postgresql for seamless data persistence.
• quarkus-smallrye-reactive-messaging and quarkus-kafka-client for reactive messaging and Apache Kafka integration.
• quarkus-oidc for implementing OpenID Connect authentication and securing applications.
• quarkus-micrometer and quarkus-opentelemetry for observability, metrics, and tracing.
• quarkus-container-image-docker and quarkus-kubernetes for containerization and deployment to Kubernetes platforms.

These extensions are widely adopted because they reduce boilerplate, provide reliable default configurations out of the box, and follow cloud-native best practices—making it easy to plug Quarkus into real-world architectures.

“OK, I would like to try it, but is it easy enough to migrate my workflow to Quarkus?”

Migrating to a new framework can feel daunting, even when it promises better performance, lower costs, and an improved developer experience. It’s like being offered a better house in a better neighborhood, but hesitating because of the hassle of packing, moving, and settling in.

With Quarkus, the transition doesn’t have to be disruptive. Thanks to its compatibility with standard Java APIs, support for Jakarta EE and Spring, and a wide range of extensions, many projects can adopt Quarkus incrementally without rewriting existing code. Whether you’re coming from a traditional Java EE application server, a Spring-based stack, or another framework such as Micronaut or Dropwizard, Quarkus provides familiar APIs, tooling, and migration guides to ease the transition.

The platform supports commonly used Jakarta specifications like JAX-RS, CDI, JPA, and Bean Validation out of the box. For Spring users, the compatibility layer includes support for widely used annotations and components. See the Spring DI guide to learn more.

Need assistance getting started? You’re not alone. The Quarkus team offers expert guidance throughout the migration journey, from initial architecture reviews to production readiness. Whether you’re evaluating the framework or planning a full transition, support is available to help ensure a smooth and successful adoption.

All it takes is a decision to move forward. Your team deserves a faster, leaner, and cloud-native future.

Concluding note

Quarkus is redefining Java development by combining modern features with the robustness of the Java ecosystem.

Its focus on developer productivity, performance, and seamless integration positions it as a formidable framework for building efficient, cloud-native applications.

Whether you’re looking to optimize costs, enhance development speed, or adopt a reactive approach, Quarkus is a game-changer for Java developers.

Do you want to know more? Keep reading the second article in the series:

Quarkus: A Runtime and Framework for Cloud-Native Java

The post Optimizing Java for the Cloud-Native Era with Quarkus appeared first on foojay.

In my previous article, I outlined a comprehensive list of features introduced in Spring Framework 7 and Spring Boot 4.

In this series of articles, we will explore these features in detail using a pragmatic approach. In this second article, I will dive deep into integration of Observability framework i.e., OpenTelemetry with SpringBoot.

As an architect and developer, when I engage in system design—whether it involves monolithic architecture, microservices, or contemporary cloud-native applications—I have made the integration of observability patterns utilizing open-telemetry tools a standard practice. Through observability, we can monitor application behavior via metrics, logs, and traces that we trigger

Before the release of Spring Boot 4, developers needed to incorporate numerous dependencies related to open telemetry, including various micrometer dependencies, which could be quite overwhelming at times. However, with the introduction of Spring Boot 4, the team streamlined this process by adding a single starter dependency, namely spring-boot-starter-opentelemetry, which automatically includes most of the micrometer dependencies. The OTLP protocol serves as the key enabler here, rather than any specific library.

Before we deep dive into OpenTelemetry with Spring Boot 4, let us first grasp some fundamental concepts and understand what each of them signifies.

Key Terminology

• Metrics act as numerical representations of aggregated data that various input sources, including hardware, software, and applications, provide. This includes monitoring resource utilization, performance, and user behavior. Teams can utilize various types of metrics, specifically application metrics, system metrics, and business metrics. Mainly talks about what happens inside the system
• Logs provide granular information regarding the specifics of the events, offering deeper insights into the reasons behind issues that occur at specific timestamps. Mainly talks about why issues occur
• Traces show us the events that occur during a complete transaction of a request or session, allowing us to pinpoint where and what transpired within a user session. By utilizing traces, we can identify which metrics or logs we associate with a specific issue, thereby helping us mitigate future problems such as API request slowness, API traffic, service-to-service interactions, workload, and internal API calls. Mainly talks about where the issue is occurring.
• Observability is evaluating the current state of an application and involves integrating logs, metrics, and traces data.
• Telemetry involves collecting all the unprocessed data that contributes to monitoring and enables more in-depth analysis, yielding thorough insights.
• The OpenTelemetry documentation states that OpenTelemetry is an open-source framework for observability that facilitates users in generating, exporting, and collecting telemetry data such as logs, metrics, and traces.
• Spring Boot Actuator, a subproject of Spring Boot, facilitates the management and monitoring of our application through HTTP endpoints or JMX. It reveals multiple endpoints that provide extensive information about the application across various instrumentation details.

After we understand the key terminology, let’s explore how and what changed in the integration of open telemetry with Spring Boot 4.

Why OpenTelemetry?

In contemporary cloud-native applications, we will develop a sophisticated microservice architecture that integrates intricate business requirements. We find the necessity for both application and infrastructure observability to be crucial.

The OpenTelemetry framework guides us with two fundamental principles:

• You own the data that you create, ensuring there is no vendor lock-in.
• You need to familiarize yourself with only one set of APIs and conventions.

Before the release of Spring Boot 4, we faced several challenges in integrating open telemetry, specifically:

• OpenTelemetry Java Agent—Zero code changes but has version compatibility issues.
• 3rd-party OpenTelemetry Spring Boot Starter - External dependency

With Spring Boot 4, developers provide open telemetry with either native or in-house support as a starter dependency. This spring-boot-starter-opentelemetry is the recommended approach, as it

• Supports GraalVM native-image and AOT compilation
• Integrates natively with Micrometer
• Exports signals via OTLP protocol
• Works seamlessly with Spring Boot 4.0

Step-by-Step Guide

If you already have an existing Spring Boot application, you can begin by incorporating the following dependency; otherwise, you can start creating a Spring Boot application using Spring Initializr.

Step 1: Add the opentelemetry dependency in the pom.xml

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-opentelemetry</artifactId>
</dependency>

Or in the build.gradle

implementation 'org.springframework.boot:spring-boot-starter-opentelemetry'

Note: This starter includes

• OpenTelemetry API
• io.micrometer:micrometer-registry-otlp for metrics
• io.micrometer:micrometer-tracing-bridge-otel for traces

In Spring Boot 3, you must add the dependencies mentioned above individually to implement OpenTelemetry.

Step 2: Configure Logs Export

1. Add the OpenTelemetry Logback appender to pom.xml

<dependency>
    <groupId>io.opentelemetry.instrumentation</groupId>
    <artifactId>opentelemetry-logback-appender-1.0</artifactId>
    <version>2.21.0-alpha</version> // version might change later
</dependency>

Or in build.gradle:

implementation 'io.opentelemetry.instrumentation:opentelemetry-logback-appender-1.0:2.21.0-alpha'

2. Enable Log Export Property by adding the below property in application.properties

management.opentelemetry.logging.export.otlp.endpoint=http://localhost:4318/v1/logs

Or in YAML:

management:
  opentelemetry:
    logging:
      export:
        otlp:
          endpoint: http://localhost:4318/v1/logs

3. Configure Logback Appender

Create src/main/resources/logback-spring.xml:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <include resource="org/springframework/boot/logging/logback/base.xml"/>

    <appender name="OTEL" class="io.opentelemetry.instrumentation.logback.appender.v1_0.OpenTelemetryAppender">
    </appender>

    <root level="INFO">
        <appender-ref ref="OTEL"/>
    </root>
</configuration>

4. Install OpenTelemetry Appender

Create a component to initialize the appender:

import io.opentelemetry.api.OpenTelemetry;
import io.opentelemetry.instrumentation.logback.appender.v1_0.OpenTelemetryAppender;
import org.springframework.beans.factory.InitializingBean;
import org.springframework.stereotype.Component;

@Component
public class InstallOpenTelemetryAppender implements InitializingBean {

    private final OpenTelemetry openTelemetry;

    public InstallOpenTelemetryAppender(OpenTelemetry openTelemetry) {
        this.openTelemetry = openTelemetry;
    }

    @Override
    public void afterPropertiesSet() throws Exception {
        OpenTelemetryAppender.install(openTelemetry);
    }
}

Step 3: Configure Metrics Export

1. Enable Metrics Export

Add the following property to application.properties:

# For development keep 1.0 for all traces to export, production environment keep 0.1
management.tracing.sampling.probability=1.0
management.otlp.metrics.export.url=http://localhost:4318/v1/metrics

Or in application.yml:

management:
  otlp:
    metrics:
      export:
        url: http://localhost:4318/v1/metrics

2. Configure OpenTelemetry Semantic Conventions

You can select the metrics for export and configure them as beans, such as Process Metrics, JVM Memory Metrics, and JVM Thread Metrics.

Create a configuration class to use OpenTelemetry semantic conventions:

import io.micrometer.core.instrument.Tags;
import io.micrometer.core.instrument.binder.jvm.ClassLoaderMetrics;
import io.micrometer.core.instrument.binder.jvm.JvmMemoryMetrics;
import io.micrometer.core.instrument.binder.jvm.JvmThreadMetrics;
import io.micrometer.core.instrument.binder.jvm.convention.otel.OpenTelemetryJvmClassLoadingMeterConventions;
import io.micrometer.core.instrument.binder.jvm.convention.otel.OpenTelemetryJvmCpuMeterConventions;
import io.micrometer.core.instrument.binder.jvm.convention.otel.OpenTelemetryJvmMemoryMeterConventions;
import io.micrometer.core.instrument.binder.jvm.convention.otel.OpenTelemetryJvmThreadMeterConventions;
import io.micrometer.core.instrument.binder.system.ProcessorMetrics;

@Configuration(proxyBeanMethods = false)
public class OpenTelemetryConfiguration {

    @Bean
    OpenTelemetryServerRequestObservationConvention openTelemetryServerRequestObservationConvention() {
        return new OpenTelemetryServerRequestObservationConvention();
    }

    @Bean
    OpenTelemetryJvmCpuMeterConventions openTelemetryJvmCpuMeterConventions() {
        return new OpenTelemetryJvmCpuMeterConventions(Tags.empty());
    }

    @Bean
    ProcessorMetrics processorMetrics() {
        return new ProcessorMetrics(
            List.of(), 
            new OpenTelemetryJvmCpuMeterConventions(Tags.empty())
        );
    }

    @Bean
    JvmMemoryMetrics jvmMemoryMetrics() {
        return new JvmMemoryMetrics(
            List.of(), 
            new OpenTelemetryJvmMemoryMeterConventions(Tags.empty())
        );
    }

    @Bean
    JvmThreadMetrics jvmThreadMetrics() {
        return new JvmThreadMetrics(
            List.of(), 
            new OpenTelemetryJvmThreadMeterConventions(Tags.empty())
        );
    }

    @Bean
    ClassLoaderMetrics classLoaderMetrics() {
        return new ClassLoaderMetrics(
            new OpenTelemetryJvmClassLoadingMeterConventions()
        );
    }
}

Step 4: Configure Traces Export

1. Add the following property to enable trace export in application.properies

management.opentelemetry.tracing.export.otlp.endpoint=http://localhost:4318/v1/traces

Or in application.yml

management:
  opentelemetry:
    tracing:
      export:
        otlp:
          endpoint: http://localhost:4318/v1/traces

Spring Boot automatically configures:

• OtlpHttpSpanExporter (or OtlpGrpcSpanExporter for gRPC)
• OpenTelemetry SDK for trace export
• Micrometer Observation API to OpenTelemetry bridge

In the configuration mentioned above, management.otlp.metrics.export.url specifies the destination for metrics in Spring Boot which uses Micrometer's OTLP exporter (hence management.otlp.metrics). A collector that is compatible with OTLP directs the data. management.opentelemetry.tracing.export.otlp.endpoint indicates where Spring Boot should send traces and use the OpenTelemetry tracing bridge (hence management.opentelemetry.tracing).

Both send data to the same collector (port 4318), but the configuration paths differ because of the way the libraries integrate.

Step 5: Add Trace ID to HTTP Response Headers

Create a Servlet filter to include trace ID in response headers:

import io.micrometer.tracing.Tracer;
import jakarta.servlet.FilterChain;
import jakarta.servlet.ServletException;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import org.jspecify.annotations.Nullable;
import org.springframework.stereotype.Component;
import org.springframework.web.filter.OncePerRequestFilter;

import java.io.IOException;
@Component
public class TraceIdFilter implements Filter {

    @Override
    public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) 
            throws IOException, ServletException {

        if (response instanceof HttpServletResponse httpResponse) {
            String traceId = Span.current().getSpanContext().getTraceId();
            httpResponse.setHeader("X-Trace-Id", traceId);
        }

        chain.doFilter(request, response);
    }
}

This allows users to include the trace ID when reporting errors.

Testing with Docker Compose

1. Set Up Local OTLP Backend

Create compose.yaml in your project root:

services:
  otel-lgtm:
    image: grafana/otel-lgtm
    ports:
      - "3000:3000"   # Grafana UI
      - "4318:4318"   # OTLP HTTP receiver
      - "4317:4317"   # OTLP gRPC receiver
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin

2. Add Docker Compose Support (Optional)

Add dependency for automatic Docker Compose integration:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-docker-compose</artifactId>
    <scope>runtime</scope>
    <optional>true</optional>
</dependency>

Spring Boot will automatically:

• Detect compose.yaml
• Run docker compose up
• Configure OTLP endpoints automatically

3. Run and Verify

1. Start your application
2. Access Grafana at http://localhost:3000 (username: admin, password: admin)
3. Make requests to your application
4. View:
   • Logs: All application logs with trace IDs
   • Traces: Distributed traces with span details
   • Metrics: JVM metrics, custom metrics, HTTP metrics
5. Navigate Home > Explore > Tempo
6. Click Search and select service "spring-boot-app-with-mongodb"
7. Click on a Trace to see the span details

The complete code can be found over on Github.

Conclusion

Developers must incorporate observability into modern cloud-native applications, as it has become essential and significantly aids in effectively monitoring the application. With spring-boot-starter-opentelemetry dependency, spring boot it automcatically perform an instrumentation for:

• The HTTP server handles requests for all controller endpoints.
• The HTTP client makes requests using RestTemplate(deprecated), RestClient, and WebClient.
• The JDBC database executes calls.
• Logs contain Trace/span IDs.

References

1. OpenTelemetry: https://opentelemetry.io/docs/what-is-opentelemetry/
2. Observability https://www.elastic.co/blog/observability-metrics
3. Spring Blog https://spring.io/blog/2025/11/18/opentelemetry-with-spring-boot

Happy Learning!

The post Spring Boot 4 OpenTelemetry Guide: Metrics, Traces, and Logs Explained appeared first on foojay.

👨‍💻 GitHub: https://github.com/vinny59200/dukeburger

🔵⚪⚪⚪⚪⚪⚪⚪⚪⚪⚪⚪

TL;DR

This guide shows how to use Micrometer and Prometheus in Spring Boot to track a custom metric for a Kafka-driven Burger Orders app. You’ll post a burger order to a REST endpoint, publish it to Kafka, consume the topic, and increment a counter for all “DukeBurger” orders. Copy the snippets, run, and you’ll see your metric on /actuator/prometheus.

🔵🔵⚪⚪⚪⚪⚪⚪⚪⚪⚪⚪

Why Micrometer and Prometheus?

Micrometer is a vendor-neutral metrics facade. Your code records counters, timers, and gauges once; Micrometer ships those to many backends (Prometheus, Datadog, etc.) via simple registries. Prometheus is a time-series database that pulls metrics by scraping an HTTP endpoint periodically (Spring exposes /actuator/prometheus). Micrometer Application Observability

Key ideas:

• Micrometer offers a simple API: Counter, Timer, Gauge.
• Spring Boot Actuator autoconfigures Micrometer and exposes metrics endpoints, including Prometheus format. See
• Prometheus “scrapes,” so your app just exposes a text endpoint—no push needed. docs.micrometer.io

🔵🔵🔵⚪⚪⚪⚪⚪⚪⚪⚪⚪

What the Burger Orders App Does

1. Order a burger via HTTP POST /orders?burger=DukeBurger.
2. Produce an Avro message to Kafka topic burger.orders.
3. Consume burger.orders with @KafkaListener.
4. Increment a Micrometer Counter named events_DukeBurger_total whenever the burger is "DukeBurger".
5. Expose metrics at /actuator/prometheus for Prometheus to scrape.

This pattern is common: REST → Kafka → Consumer → Metric. Spring Kafka makes producing and consuming concise; Micrometer makes metrics easy. See

🔵🔵🔵🔵⚪⚪⚪⚪⚪⚪⚪⚪

The Data Contract (Avro)

{
  "type": "record",
  "name": "BurgerOrder",
  "namespace": "com.vv.burger",
  "fields": [
    { "name": "burger", "type": "string" },
    { "name": "timestamp", "type": "string" }
  ]
}

Why: A tiny schema keeps the demo clear. Avro gives you compact messages and generated classes.

🔵🔵🔵🔵🔵⚪⚪⚪⚪⚪⚪⚪

Hot Spots: Minimal Code You Need

Spring initializer for Micrometer & Prometheus in Spring Boot: Kafka Burger Orders

1) Expose a Counter with Tags (Micrometer)

package com.vv.burger.config;

import io.micrometer.core.instrument.Counter;
import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.Tag;
import io.micrometer.core.instrument.Tags;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class MetricsConfig {

    @Bean
    public Counter burgerOrderCounter(MeterRegistry registry) {
        // Common tags for the burger app
        Tags tags = Tags.of(
                Tag.of("app", "burger-service"),
                Tag.of("topic", "burger.orders")
                           );

        return Counter.builder("events_DukeBurger_total")
                      .description("Count of DukeBurger order events processed")
                      .baseUnit("orders")
                      .tags(tags)
                      .register(registry);
    }
}

Side note: We add consistent tags now (app, topic) so you can filter and graph later. See

🔵🔵🔵🔵🔵🔵⚪⚪⚪⚪⚪⚪

2) REST Controller → Produce to Kafka

package com.vv.burger.controller;

import com.vv.burger.BurgerOrder;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

import java.nio.charset.StandardCharsets;
import java.time.OffsetDateTime;
import java.util.UUID;

@RestController
@RequestMapping( "/orders" )
public class OrderController {

    private final KafkaTemplate<String, BurgerOrder> kafkaTemplate;
    private final String                             topic;

    public OrderController( KafkaTemplate<String, BurgerOrder> kafkaTemplate,
                            @Value( "${app.kafka.topic}" ) String topic ) {
        this.kafkaTemplate = kafkaTemplate;
        this.topic = topic;
    }

    @PostMapping
    public String sendOrder( @RequestParam String burger ) {
        // 1. Build the Avro payload (BurgerOrder must be a generated Avro class)
        BurgerOrder order = BurgerOrder.newBuilder()
                                       .setBurger( burger )
                                       .setTimestamp( OffsetDateTime.now()
                                                                    .toString() )
                                       .build();

        // 2. Create CloudEvent metadata as headers
        String id = UUID.randomUUID()
                        .toString();
        OffsetDateTime now = OffsetDateTime.now();

        ProducerRecord<String, BurgerOrder> record = new ProducerRecord<>( topic, order );
        record.headers()
              .add( "ce_id", id.getBytes( StandardCharsets.UTF_8 ) );
        record.headers()
              .add( "ce_type", "BurgerOrder".getBytes( StandardCharsets.UTF_8 ) );
        record.headers()
              .add( "ce_source", "http://localhost/orders".getBytes( StandardCharsets.UTF_8 ) );
        record.headers()
              .add( "ce_specversion", "1.0".getBytes( StandardCharsets.UTF_8 ) );
        record.headers()
              .add( "ce_time", now.toString()
                                  .getBytes( StandardCharsets.UTF_8 ) );
        record.headers()
              .add( "ce_subject", "order".getBytes( StandardCharsets.UTF_8 ) );
        record.headers()
              .add( "ce_datacontenttype", "application/avro".getBytes( StandardCharsets.UTF_8 ) );

        // 3. Send the record
        kafkaTemplate.send( record );

        return "✅ Order sent to Kafka: " + burger;
    }
}

Side note: The headers mimic CloudEvents so you can plug into event tooling later. This is optional for the metric. Cloud Events

🔵🔵🔵🔵🔵🔵🔵⚪⚪⚪⚪⚪

3) Kafka Consumer → Count “DukeBurger”

package com.vv.burger.consumer;

import com.vv.burger.BurgerOrder;
import io.cloudevents.CloudEvent;
import io.cloudevents.core.builder.CloudEventBuilder;
import io.micrometer.core.instrument.Counter;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.springframework.context.annotation.Bean;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory;
import org.springframework.kafka.core.ConsumerFactory;
import org.springframework.stereotype.Component;

import java.net.URI;
import java.time.OffsetDateTime;
import java.util.Map;

@Component
public class ConsumerApp {

    // Injects the Counter bean defined in MetricsConfig (events_DukeBurger_total)
    private final Counter burgerOrderCounter;

    public ConsumerApp( final Counter burgerOrderCounter ) {
        this.burgerOrderCounter = burgerOrderCounter;
    }

    @KafkaListener( topics = "burger.orders",
                    groupId = "group1" )
    public void receive( ConsumerRecord<String, BurgerOrder> record ) {
        BurgerOrder order = record.value();

        // Optionally reconstruct CloudEvent from headers
        CloudEvent cloudEvent = CloudEventBuilder.v1()
                                                 .withId( getHeader( record, "ce_id" ) )
                                                 .withType( getHeader( record, "ce_type" ) )
                                                 .withSource( URI.create( getHeader( record, "ce_source" ) ) )
                                                 .withSubject( getHeader( record, "ce_subject" ) )
                                                 .withTime( OffsetDateTime.parse( getHeader( record, "ce_time" ) ) )
                                                 .withDataContentType( getHeader( record, "ce_datacontenttype" ) )
                                                 .withData( "application/avro", order.toString()
                                                                                     .getBytes() ) // optional
                                                 .build();

        System.out.println( "📥 Received order: " + order.getBurger() + " at " + order.getTimestamp() );
        System.out.println( "🧾 CloudEvent type: " + cloudEvent.getType() + ", id: " + cloudEvent.getId() );

        if ( isDukeBurger( order ) ) {
            burgerOrderCounter.increment();
        }
    }

    private boolean isDukeBurger( final BurgerOrder order ) {
        return "DukeBurger".equals( order.getBurger()
                                         .toString() );
    }

    private String getHeader( ConsumerRecord<?, ?> record, String key ) {
        return new String( record.headers()
                                 .lastHeader( key )
                                 .value() );
    }

    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, BurgerOrder> kafkaListenerContainerFactory() {
        ConcurrentKafkaListenerContainerFactory<String, BurgerOrder> factory =
                new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory( consumerFactory() );
        return factory;
    }

    public ConsumerFactory<String, BurgerOrder> consumerFactory() {
        Map<String, Object> props = Map.of(
                "bootstrap.servers", "kafka:9092",
                "group.id", "group1",
                "key.deserializer", StringDeserializer.class.getName(),
                "value.deserializer", io.confluent.kafka.serializers.KafkaAvroDeserializer.class.getName(),
                "schema.registry.url", "http://schema-registry:8081",
                "specific.avro.reader", true
                                          );

        return new org.springframework.kafka.core.DefaultKafkaConsumerFactory<>( props );
    }
}

Side note: @KafkaListener binds the method to the topic with minimal boilerplate. Keep consumer config small for a first run. See

🔵🔵🔵🔵🔵🔵🔵🔵⚪⚪⚪⚪

4) Avro Bytes → Object (utility)

package com.vv.burger.consumer;

import com.vv.burger.BurgerOrder;
import org.apache.avro.io.BinaryDecoder;
import org.apache.avro.io.DecoderFactory;
import org.apache.avro.specific.SpecificDatumReader;

import java.io.ByteArrayInputStream;

public class AvroUtils {
    public static BurgerOrder fromBytes( byte[] bytes ) {
        try ( ByteArrayInputStream in = new ByteArrayInputStream( bytes ) ) {
            SpecificDatumReader<BurgerOrder> reader = new SpecificDatumReader<>( BurgerOrder.class );
            BinaryDecoder decoder = DecoderFactory.get()
                                                  .binaryDecoder( in, null );
            return reader.read( null, decoder );
        } catch ( Exception e ) {
            throw new RuntimeException( "Failed to deserialize BurgerOrder Avro event", e );
        }
    }
}

Side note: Spring Kafka + Confluent deserializer already returns BurgerOrder, so you rarely need this. It’s useful in tests or when you manually handle bytes.

🔵🔵🔵🔵🔵🔵🔵🔵🔵⚪⚪⚪

Application Properties (essentials)

spring:
  application:
    name: burger-service

# Kafka
app:
  kafka:
    topic: burger.orders

# Actuator + Micrometer Prometheus
management:
  endpoints:
    web:
      exposure:
        include: health,info,metrics,prometheus
  endpoint:
    health:
      show-details: always

Side note: This exposes /actuator/prometheus so Prometheus can scrape. See

🔵🔵🔵🔵🔵🔵🔵🔵🔵🔵⚪⚪

Run & Observe

1. Build the image : docker build -t my-spring-boot-app:latest .
2. Run the app: docker-compose up -d
3. Create the topic: http://localhost:8080/ui/clusters/local/all-topics/create-new-topic named burger.orders
4. Send a few orders:
   • curl -X POST "http://localhost:8080/orders?burger=DukeBurger
   • curl -X POST "http://localhost:8080/orders?burger=Veggie
   • curl -X POST "http://localhost:8080/orders?burger=DukeBurger
5. Check metrics: open http://localhost:8080/actuator/prometheus and search for events_DukeBurger_total. You should see it increase after each “DukeBurger” consumed.
6. Check in JMC: Connect  JMC to your app and In JMC; open MBean Browser (left pane); Expand the metric ; Navigate to the counter events_DukeBurger_total; Click it → Attributes tab → read Count; You should see it increase after each “DukeBurger” consumed.

JMC for Micrometer & Prometheus in Spring Boot: Kafka Burger Orders

🔵🔵🔵🔵🔵🔵🔵🔵🔵🔵🔵⚪

Takeaways

• Small steps win: REST → Kafka → Consumer → Metric is a powerful, simple pipeline.
• Micrometer first: Write metrics once; swap backends later (Prometheus today, Datadog tomorrow). See
• Tags matter: Add app and topic tags now. Your future dashboards will thank you.
• Avro stays lean: A tiny schema keeps payloads small and generated classes easy to use.
• CloudEvents optional: The headers help interoperability but are not required for Micrometer. Cloud Events

🔵🔵🔵🔵🔵🔵🔵🔵🔵🔵🔵🔵

Conclusion

You just wired Micrometer and Prometheus in Spring Boot around a Kafka flow and produced a clean, tagged counter you can graph and alert on. From here, extend the metric set (timers for latency, gauges for queue depth), add dashboards, and create an alert when events_DukeBurger_total stalls or spikes.

Further Reading (Foojay)

• Spring Boot + Kafka Streams testing & routing:
  https://foojay.io/today/spring-boot-kafka-streams-event-routing-testing/
• Make Spring Batch fly with native image & GraalVM:
  https://foojay.io/today/speed-up-your-spring-batch-with-native-image-and-graalvm/

Recommended Courses (go further with certification)

• Java OCP prep (Udemy):
  https://www.udemy.com/course/ocp-oracle-certified-professional-java-developer-prep/?referralCode=54114F9AD41F127CB99A
• Spring Professional – 6 full tests (Udemy):
  https://www.udemy.com/course/spring-professional-certification-6-full-tests-2v0-7222-a/?referralCode=04B6ED315B27753236AC
• Spring Certification Book (Leanpub/ Kindle/ Paperback):
  https://spring-book.mystrikingly.com/

References

• Micrometer docs and Prometheus registry overview. Micrometer Application Observability+1
• Spring Boot Actuator metrics. See
• @KafkaListener reference. See
• CloudEvents Java SDK. Cloud Events

The post Micrometer & Prometheus in Spring Boot: Kafka Burger Orders🍔📨 appeared first on foojay.

Last week, I described several approaches to OpenTelemetry on the JVM, their requirements, and their different results. This week, I want to highlight several gotchas found across stacks in the zero-code instrumentation.

The promise of OpenTelemetry

Since its inception, OpenTelemetry has unified the 3 pillars of observability. In the distributed tracing space, it replaced proprietary protocols Zipkin and Jaeger. IMHO, it achieved such success for several reasons:

• First, a huge industry pressure to work across proprietary tools
• Zero-code instrumentation, allowing developers to be unconcerned by OpenTelemetry
• Easy and unified configuration mechanism via environment variables.

The latter is a boon for the Ops team, as they don't have to know the underlying framework (or stack!) details. They only need to check the Environment Variable Specification and they are done.

Here's a simple snippet to illustrate my point:

environment:
  OTEL_EXPORTER_OTLP_ENDPOINT: http://collector:4317             #1
  OTEL_SERVICE_NAME: foobar                                      #2

1. Configure the endpoint to send data to
2. Set the component's name

From the above, you can't guess what the underlying stack is.

On the OpenTelemetry side, different developers contribute to different language libraries. Developers of the Java agent are different from the those of the Python library. Moreover, each stack has specific approaches and limitations.

It naturally creates differences in the different implementations.

Gotchas

Here are a couple of gotchas I found out, but the list is not exhaustive.

Path or no path?

Let's start easy with a gotcha that exists across stacks.

OpenTelemetry offers some configuration parameters for endpoints.

• A generic one, to use for an OpenTelemetry Collector that accepts all signals: OTEL_EXPORTER_OTLP_ENDPOINT
• Parameters specialized for a single signal, e.g., OTEL_EXPORTER_OTLP_TRACES_ENDPOINT

In the latter case, one can set the root path to the endpoint, e.g, http://collector:4317. The SDK will automatically append the default path, depending on the signal type. For example, it appends /v1/traces for the OTEL_EXPORTER_OTLP_TRACES_ENDPOINT configuration parameter.

Alternatively, one can set the full path to the endpoint, e.g, http://collector:4317/v1/traces.

The gotcha will catch you when the OpenTelemetry evolves to v2 if you didn't use paths. Because the library will automatically append /v1/, you'll have to make sure the backend offers both /v1 and /v2 endpoints.

Python logging

To open the list, here's the first gotcha: by default, the Python library doesn't send logging data. It must be enabled explicitly!

environment:
  OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED: true         #1
  OTEL_EXPORTER_OTLP_ENDPOINT: http://collector:4317
  OTEL_SERVICE_NAME: foobar

1. Enable logs

Developers also must be involved:

Unlike Traces and Metrics, there is no equivalent Logs API. There is only an SDK. For Python, you use the Python logger library, and then the OTel SDK attaches an OTLP handler to the root logger, turning the Python logger into an OTLP logger. One way to accomplish this is documented in the logs example in the OpenTelemetry Python repository.

-- Logs Auto-Instrumentation Example

Micrometer Tracing

Before OpenTelemetry, Jaeger and Zipkin reigned supreme in the distributed tracing area. In the great Spring tradition, the project created Spring Cloud Sleuth to offer a facade over Zipkin. Over time, it evolved to be compatible with OpenTracing, one of OpenTelemetry's parents, along with OpenCensus.

I cannot be sure whether OpenTelemetry was the cause, but Spring Cloud Sleuth evolved into the (badly-named) Micrometer Tracing library. Micrometer Tracing targets Zipkin first, but also supports OpenTelemetry. I find adding the requested libraries a bit complex, but it's manageable.

However, the configuration doesn't conform to the OpenTelemetry environment variables. It brings its variables:

environment:
  MANAGEMENT_OTLP_TRACING_ENDPOINT: http://jaeger:4318/v1/traces #1-2
  OTEL_SERVICE_NAME: foobar                                      #3

1. Non-OpenTelemetry environment variable name
2. MUST set the full path
3. Conform to the OpenTelemetry spec since Spring Boot 3.5. Before it, it used spring.application.name

Quarkus

Compare with Quarkus, which prefixes regular OpenTelemetry environment variables with QUARKUS_:

environment:
  QUARKUS_OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: http://jaeger:4317
  QUARKUS_OTEL_SERVICE_NAME: foobar

It's consistent. And yet, the Quarkus instrumentation has another gotcha:

Only the tracing signal is enabled by default. To enable metrics and logs, add the following configuration to your application.properties file:

-- Quarkus instrumentation

Meanwhile, if you're an expert at operating OpenTelemetry, these defaults will surprise you. In OpenTelemetry, they are governed by other configuration parameters:

environment:
  OTEL_METRICS_EXPORTER: none
  OTEL_LOGS_EXPORTER: none

Summary

OpenTelemetry has become a de facto standard in a few years. However, the promise of ubiquitous configuration, if there was ever such a thing, doesn't hold. Operators of OpenTelemetry can't treat services as black boxes. They must consider the underlying stack and framework, and learn how to configure them accordingly.

On the JVM, I'd recommend to stick to the Java Agent as much as possible, but if you must across stacks, there's no magical solution.

To go further:

• Maven - Introduction to the Dependency Mechanism
• Effective Rust - Item 25: Manage your dependency graph

Originally published at A Java Geek on August 10^th, 2025

The post OpenTelemetry configuration gotchas appeared first on foojay.

What are the Application Observability Code Challenges?GoalsWhat do the code challenges look like?Your own stackOnline environment(s)The first challenge How to Get StartedChallenge details

Code challenges are a nice way to challenge yourself with programming, resulting in some great challenges like #1brc and the yearly Advent of Code (#AoC).

While these challenges often include a competitive element, the Application Observability Code Challenges are focussed on the learning about observability practices.

What are the Application Observability Code Challenges?

The idea of these challenges came to me based on the challenges above and the fact that I see in practice that quite a few developers are struggling to get up to speed with applying observability practices.

The challenges are small applications or code samples. When you normally write your code, you build the application with a goal in mind and you test the behaviour of your application.

But do you know that your application does what it is supposed to do? Observability is about:

How effectively you can understand the behaviour of the system from the outside, using the data it generates.

The sample application has a certain behaviour, and can you understand this behaviour from outside the system?

If you put your own application into production and you get paged in the middle of the night because of some problems, how do you know for sure that the system is behaving as expected?

This requires two things: first, that your application is observable, and second, that you have the skills to use your observability tools.

But also during development, when you run your (performance) tests, you need to understand the behaviour of your application to make the right decisions about whether your code changes are production-ready.

Observability data can help you make that decision.

The goal of these code challenges is to practice and gain more experience in both making your application observable and using the observability tools.

Goals

The goals of the challenges:

• 🎉 - Have fun !
• 🖵 - Learn to understand the behaviour of the code
• 📈 - Learn how to use observability tools to understand code behaviour 📈
• 🔍 - Spot the unexpected behaviour!
• 🤗 - Practice and learn!
• 🎁 - Share your findings and solution, either as a comment or as a pull request

The goal is not to discuss the libraries, frameworks or specific code implementation used, but to practice and learn!

What do the code challenges look like?

My background is in Java development, so most of the challenges I will provide will be Java challenges. I have a number of them in mind.

The challenge will be available in a git repository, all prepared to run on your system as smooth as possible.

The application will already produce some telemetry data with OpenTelemetry.

The challenge is to find the problem and extend the observability to get better insights and also proof that a potential solution works as expected.

Using the Observability Toolkit you can easily spin up OpenTelemetry and Grafana based observability tools.

You can then run a sample application and a test script.

In Grafana you can see the first results, then it is up to you to continue. Some hints will be provided.

The whole setup will look like this:

Your own stack

If you usually use other observability tools, I encourage you to use them!

The sample application is prepared to send data using OpenTelemetry, so any setup that supports OTLP will work.

Online environment(s)

Besides running it locally, I am also preparing a guided online environment with Killercoda, https://killercoda.com/observability-code-challenges.
That way you don't have to mess with your local machine and it will give you some more guidance.

I may also add a setup with devcontainers so you can easily run it in other ways. Please let me know if you are interested.

The first challenge

The steps to follow for the first challenge:

• Run the sample application
• Run the tests to see what happens
• Try to find out what happens, make a hypothesis❗
• Improve the observability of the application to prove the hypothesis
• Optional: fix the problem and prove it with observability data that it is really fixed
• Optional, but highly appreciated 🙏: Share your findings, insights you learned and potential solution, either as a 'discussion' or as a pull request

🛠️ How to Get Started

All the details you need to jump in can be found here:
👉 https://github.com/cbos/application-observability-code-challenges/tree/main/challenge-01

Prefer an online environment? No problem! Use KillerCode to get started with just a few clicks:
👉 https://killercoda.com/observability-code-challenges

Challenge details

• The sample application is a simple Spring Boot application with a REST endpoint implemented in Jersey/JAX-RS.
• The application is instrumented using OpenTelemetry auto instrumentation.
• You can either run the application with Docker or directly.
• Pre-configured K6 load scripts to simulate traffic and reveal performance bottlenecks.
• You can use the pre-configured Observability Toolkit or you can use your own Observability stack

The setup looks like this:

After running one of the scripts you can get more details in a Grafana dashboard like this:

In this screenshot you can see that the application is reaching a limit for some reason, more load does not give more requests per second and with more load the response times increase a lot.

Are you able to improve the observability, find the cause and maybe even fix it?

👉 Go to the challenge: https://github.com/cbos/application-observability-code-challenges/tree/main/challenge-01

Originally published on ceesbos.nl in January 2025

The post Challenge yourself with Application Observability Code Challenges appeared first on foojay.