Multimodal Context-aware AI Agent

Files are managed using Pixeltable Tables, which store references efficiently using built-in multimodal types:

# From setup_pixeltable.py
images = pxt.create_table(
    "agents.images",
    {"image": pxt.Image, "uuid": pxt.String, "timestamp": pxt.Timestamp},
    if_exists="ignore",
)
documents = pxt.create_table(
    "agents.collection",
    {"document": pxt.Document, "uuid": pxt.String, "timestamp": pxt.Timestamp},
    if_exists="ignore",
)

Processing happens via Computed Columns, which automatically apply functions (like resizing or API calls) when data is added:

# From setup_pixeltable.py
# Generate thumbnails automatically when images are added
images.add_computed_column(
    thumbnail=pxt_image.b64_encode(
        pxt_image.resize(images.image, size=(96, 96)) # Sidebar size
    ),
    if_exists="ignore",
)

# Extract audio automatically when videos are added
videos.add_computed_column(
    audio=extract_audio(videos.video, format="mp3"),
    if_exists="ignore"
)

The core agent logic runs as a declarative workflow in the agents.tools table. Key steps involve transforming data and performing semantic searches:

1. Data Transformation (Views & Iterators): Documents are chunked on-the-fly using a View and a DocumentSplitter Iterator, avoiding data duplication:

   
   # From setup_pixeltable.py
   chunks = pxt.create_view(
       "agents.chunks",
       documents, # Base table
       iterator=DocumentSplitter.create(
           document=documents.document,
           separators="token_limit", # Split by token count
           limit=300
       ),
       if_exists="ignore",
   )
   

2. Semantic Search (Embedding Indexes): Indexes are added to tables/views to enable fast similarity search using integrated models like Sentence Transformers or CLIP:

   
   # From setup_pixeltable.py
   # Index document chunks for text search
   chunks.add_embedding_index(
       "text",
       string_embed=sentence_transformer.using(model_id=config.EMBEDDING_MODEL_ID),
       if_exists="ignore",
   )
   # Index images for cross-modal search
   images.add_embedding_index(
       "image",
       embedding=clip.using(model_id=config.CLIP_MODEL_ID),
       if_exists="ignore",
   )
   

3. Context Retrieval (Query Functions): Reusable search logic is defined using the `@pxt.query` decorator. These functions are called by computed columns in the main workflow:

   
   # From setup_pixeltable.py
   @pxt.query
   def search_documents(query_text: str):
       sim = chunks.text.similarity(query_text)
       return (
           chunks.order_by(sim, asc=False)
           .select(chunks.text, source_doc=chunks.document, sim=sim)
           .where(sim > 0.8)
           .limit(10)
       )
   

4. Workflow Orchestration: The sequence of computed columns in the agents.tools table defines the entire process declaratively. It includes:
   • Calling search functions (like search_documents shown above).
   • Using User-Defined Functions (UDFs) to run custom Python code, like fetching external data. UDFs are created with the @pxt.udf decorator:

     
     # From functions.py
     @pxt.udf
     def get_latest_news(topic: str) -> str:
         """Fetch latest news for a given topic using NewsAPI."""
         # ... implementation omitted for brevity ...
         return formatted_news
     

   • Enabling Tool Calling by registering UDFs as tools for the LLM. Pixeltable provides helpers for this:

     
     # From setup_pixeltable.py
     # Define available tools
     tools = pxt.tools(
         functions.get_latest_news,
         functions.fetch_financial_data,
         functions.search_news
     )
     
     # Pass tools to the LLM (Claude) via a computed column
     tool_agent.add_computed_column(
         initial_response=messages(
             model=config.CLAUDE_MODEL_ID,
             system=tool_agent.initial_system_prompt,
             messages=[{"role": "user", "content": tool_agent.prompt}],
             tools=tools, # Make tools available
             tool_choice=tools.choice(required=True), # Force tool use
             # ... other parameters ...
         ),
         if_exists="replace",
     )
     
     # Execute the chosen tool using invoke_tools
     tool_agent.add_computed_column(
         tool_output=invoke_tools(tools, tool_agent.initial_response),
         if_exists="replace"
     )
     

   • Calling LLMs (like Claude or Mistral) to reason, generate text, or choose tools.
   Pixeltable manages the execution order and caching automatically based on these declarative steps.

Image generation is triggered by adding a prompt to the agents.image_generation_tasks table. A Computed Column calls the DALL-E 3 API via Pixeltable's OpenAI integration:

# From setup_pixeltable.py
image_gen_tasks.add_computed_column(
    generated_image=openai.image_generations(
        prompt=image_gen_tasks.prompt,
        model=config.DALLE_MODEL_ID,
        size="1024x1024"
    ),
    if_exists="ignore",
)

Persistence is achieved using Pixeltable Tables:

• Memory Bank: Saved items are stored in agents.memory_bank. An Embedding Index enables semantic search:

  
  # From setup_pixeltable.py
  memory_bank.add_embedding_index(
      column="content",
      string_embed=sentence_embed_model, # Defined earlier
      if_exists="ignore",
  )
  

• Chat History: Conversation turns are logged in agents.chat_history, allowing retrieval of recent history or semantic search over the entire conversation (also using an embedding index).