Multimodal Attributes

Image, audio, and mixed-content messages

This document describes how multimodal content (text, images, audio) is represented in OpenInference spans.

Message Content Arrays

When a message contains multiple content items (e.g., text and images), the content is represented using the message.contents array structure with flattened attributes.

Attribute Pattern

llm.input_messages.<messageIndex>.message.contents.<contentIndex>.message_content.<attribute>

Where:

• <messageIndex> is the zero-based index of the message
• <contentIndex> is the zero-based index of the content item within the message
• <attribute> is the specific content attribute

Content Type Attributes

Each content item has a type attribute that identifies its kind:

• "text" - Text content
• "image" - Image content (URL or base64)
• "audio" - Audio content (URL or base64)

Content Types

Text Content

llm.input_messages.0.message.contents.0.message_content.type = "text"
llm.input_messages.0.message.contents.0.message_content.text = "What is in this image?"

Image Content

For images referenced by URL:

llm.input_messages.0.message.contents.1.message_content.type = "image"
llm.input_messages.0.message.contents.1.message_content.image.image.url = "https://example.com/image.jpg"

For base64-encoded images:

llm.input_messages.0.message.contents.1.message_content.type = "image"
llm.input_messages.0.message.contents.1.message_content.image.image.url = "data:image/png;base64,iVBORw0KGgo..."

Audio Content

llm.input_messages.0.message.contents.2.message_content.type = "audio"
llm.input_messages.0.message.contents.2.message_content.audio.audio.url = "https://example.com/audio.mp3"

Privacy Considerations

Hiding Images

When OPENINFERENCE_HIDE_INPUT_IMAGES is set to true:

• Image URLs in input messages will be replaced with "__REDACTED__"
• This only applies when input messages are not already completely hidden

Base64 Image Truncation

When OPENINFERENCE_BASE64_IMAGE_MAX_LENGTH is set (default: 32000):

• Base64-encoded images longer than this limit will be truncated
• The truncation preserves the data URL prefix (e.g., data:image/png;base64,)
• Only the base64 content portion is subject to the length limit

Hiding Text Content

When OPENINFERENCE_HIDE_INPUT_TEXT is set to true:

• Text content in multimodal messages will be replaced with "__REDACTED__"
• This only applies when input messages are not already completely hidden

Example: Multimodal Message

A user message with both text and image content:

{
  "llm.input_messages.0.message.role": "user",
  "llm.input_messages.0.message.contents.0.message_content.type": "text",
  "llm.input_messages.0.message.contents.0.message_content.text": "What objects do you see in this image?",
  "llm.input_messages.0.message.contents.1.message_content.type": "image",
  "llm.input_messages.0.message.contents.1.message_content.image.image.url": "https://example.com/photo.jpg"
}

Fallback for Simple Messages

When a message contains only text content (no multimodal content), it can use the simpler format:

{
  "llm.input_messages.0.message.role": "user",
  "llm.input_messages.0.message.content": "Hello, how are you?"
}

The message.content attribute is used for simple text-only messages, while message.contents is used for multimodal messages.

Privacy Configuration Example

Python

from openinference.instrumentation import TraceConfig

config = TraceConfig(
    hide_input_images=True,  # Redact image URLs
    base64_image_max_length=10000,  # Limit base64 images to 10KB
    hide_input_text=False  # Keep text content
)