Table of Contents

Executive Summary: Bridging the Last Mile

Key Technical Achievements

The 5-Agent Orchestration Map

Coach Agent Workflow: Query to Answer

• 1. Context: Coach loads User Preferences (Persona, Comfort Profile) via memory tools
• 2. Intent: Coach calls Activity Agent (A2A) to classify the user's plan
• 3. Data: Coach calls Weather Agent (A2A) to get forecast (using caching)
• 4. Logic: Coach calls Stylist Agent (A2A) with all context to generate outfit plan
• 5. Safety Check: Coach calls Safety Agent (A2A) to check for risks
• 6. Response: Coach merges outfit, safety note, and style into final conversational response

Agent Deep Dive: Coach Agent (Orchestrator)

Role: The Central Nervous System - Main user interface, coordination, personalization

Architecture Security: Agent Identity

The Risk: Confused Deputy Problem

In monolithic systems, a privileged agent can be tricked into performing unauthorized actions. This risk is amplified in agents because tool execution is a trusted action.

Mitigation: Least-Privilege A2A

The A2A architecture assigns restricted Agent Identities to every service, following the Principle of Least Privilege.

Agent Deep Dive: Weather Agent (Data Specialist)

Role: Data Provider & Caching - Isolates external API calls and manages performance

Agent Deep Dive: Stylist & Activity Agents

Agent Deep Dive: Safety Agent (Guardrails)

Role: Responsible AI Implementation - Ensures safety guidelines

Safety Thresholds

Tools as Hands: Custom Functions

Tools embody discrete, testable business logic and integrate core optimizations like caching.

Feature Deep Dive: Smart Caching

Mechanism

The get_weather_smart tool uses an in-memory dictionary cache keyed by city and time_window. Data is only considered valid for 15 minutes (900 seconds).

Impact

• MLOps Focus: Demonstrates efficiency and cost control
• Cost Reduction: Weather Agent avoids unnecessary API calls (~70% reduction)
• Latency Improvement: Dramatically improves response time for common queries
• Code Principle: Logic encapsulated in tool layer (weather_tools.py), transparent to LLM

Tool Contract: Predictable Data Flow

The Challenge

The success of the Stylist Agent is entirely dependent on receiving clean, unambiguous Structured Content from the Weather Agent. If data is unreliable or malformed, the Stylist will hallucinate or fail.

Solution: Concise and Structured Output

Enforce reliability by making the Weather Agent deterministic with strict schema adherence.

Feature Deep Dive: Activity Classification

The Activity Agent translates free-form text into structured constraints used by the Stylist Agent's rule engine.

Example: "I'm going hiking this afternoon."

Context Engineering: Sessions & Memory

Feature Deep Dive: Persona Personalization

The Coach Agent reads the Persona from memory and injects persona-specific instructions into the Stylist Agent's prompt or final output formatting.

AgentOps: The Three Pillars of Observability

Observability Pillar 1: Structured Logging

Mechanism

Custom logging configuration (loging_confi.py) outputs structured JSON logs, making them easily searchable in Google Cloud Logging.

Observability: Metrics Implementation

The MetricsCollector instruments key components using Python decorators for automatic measurement.

Metrics Tracked

• agent_call_latency - Duration of agent executions
• agent_calls (by status: success/error)
• tool_execution_latency - Duration of tool calls
• tool_calls (by status: success/error)

Observability: Alerts (The "Act" Phase)

Alerts provide automated reflexes, maintaining stability in real-time. Programmatically defined using alert.py.

Alerts Created

Observability: The A2A Trace Log

Scenario: "What should I wear for hiking in Seattle today? I run cold."

Trace Log Insight: Proof of Optimization

Key Insight

The call_weather_agent span completed in only 125ms.

Conclusion

This low duration confirms the Smart Caching optimization was effective, resulting in a cache hit. This directly reduces dependency on the external Meteostat API and improves user-facing latency.

Production Metrics Evidence

Deployment Target: Vertex AI Agent Engine

Goal: Move from local development to scalable, production-ready microservices on Google Cloud.

Key Features

• Platform: Vertex AI Agent Engine Runtime - Managed service for ADK agents
• Services: Each agent deployed as independent service with dedicated endpoint (e.g., weather-agent-abc.a.run.app)
• Networking: Coach Agent uses remote A2A Protocol URLs for communication
• Managed State: Durable, highly available Session Storage for Coach Agent
• Built-in Observability: Auto-integration with Cloud Trace & Cloud Logging
• Simplified Deployment: Focus on agent logic, not infrastructure

Production Rationale: Why Agent Engine?

Cloud Run vs. Agent Engine

While Cloud Run offers general flexibility for stateless containers, Agent Engine was chosen for built-in enterprise features required for stateful, complex agents.

Key Agent Engine Advantages

A2A Rationale: Why Not Local Sub-Agents?

• Independent Scaling: Weather agent can scale to 10 instances (or zero) during peak hours without impacting Stylist agent
• Framework Agnostic: Weather agent could be swapped for Java-based legacy service as long as it adheres to A2A Protocol Contract (The Agent Card)
• Clean Contract: Communication limited to structured payloads (e.g., WeatherContext JSON), preventing memory pollution and side effects
• Security: Agent Identity & Least Privilege - Each agent has restricted permissions (mitigates Confused Deputy Problem)
• Resilience: Failure isolation - One agent's crash doesn't cascade to others
• Testability: Each agent can be tested independently with mock inputs

Deployment Architecture: 6-Service System

The system runs as a fully integrated ADK Multi-Agent System with 6 services:

Testing & Verification

Verified all 5 system components (Tools, Agents, App, Schemas, Memory) function independently before deployment.

The AgentOps Loop: Observe → Act → Evolve

Real Production Scenario

Weather Outfit ADK - Frontend

Beautiful, modern chat interface for the Weather Outfit Assistant

🎨 Features

🔌 API Endpoints

• POST /api/chat - Sends message to Coach agent (proxied to ADK Agent Engine)
• GET /health - Health check endpoint

Location-Aware Features

Quick action buttons and outfit suggestions change based on which city you search for.

Test Scenarios

👕 Outfit Icons

• T-Shirt → Shirt icon
• Jeans → Apparel icon
• Watch/Bracelet → Watch icon ⌚
• Backpack/Bag → Shopping bag icon 🛍️
• Scarf → Scatter plot icon (when cold)

Deployment Guide: 3 Methods

Alert Policies Guide (Advanced)

Problem with Simple Rate Alerts

Default alerts (e.g., > 5 errors/min) are rate-based. They don't distinguish between 5 errors in 10 requests (50% error rate) and 5 errors in 10,000 requests (0.05% error rate).

Solution: Ratio-Based Alerts (MQL)

For true production monitoring, use MQL (Monitoring Query Language) to calculate ratios. This provides accurate percentage-based alerting regardless of traffic volume.

Key Technical Learnings

• Architecture: Multi-agent systems scale better than monoliths when each agent has clear responsibility - Level 3 Collaborative Multi-Agent System is more robust and scalable
• A2A Protocol: Enables independent deployment, scaling, and language flexibility across agents with clean contracts and failure isolation
• Smart Caching: 15-minute cache TTL reduces API costs by ~70% while maintaining data freshness - Cache hits complete in 125ms vs 800ms+
• Observability: Structured logs + metrics + traces + alerts = Production confidence. P95 latency monitoring catches outlier events
• Context Engineering: Sessions (short-term) + Memory (long-term) = Personalized experiences. UserMemory class provides clean abstraction
• Safety as Code: Deterministic guardrails (Safety Agent) ensure Responsible AI compliance with hard thresholds
• Tool Design: Tools embody discrete, testable business logic. Structured output design prevents hallucination

Final System Status: Production Ready

Agent Fundamentals: Level 3 Multi-Agent System

Project Folder Structure

weather_outfit_adk/
├── app.py                    # Main ADK app entry point
├── agents/
│   ├── coach.py              # Coach orchestrator agent
│   ├── weather.py            # Weather specialist agent
│   ├── stylist.py            # Stylist recommendation agent
│   ├── activity.py           # Activity classification agent
│   └── safety.py             # Safety warning agent
├── tools/
│   ├── weather_tools.py      # Weather API + caching
│   ├── outfit_tools.py       # Outfit planning logic
│   ├── activity_tools.py     # Activity classification
│   ├── safety_tools.py       # Safety threshold rules
│   └── memory_tools.py       # User preference tools
├── schemas/
│   ├── memory.py             # UserPreferences, Persona types
│   ├── weather.py            # WeatherData schema
│   ├── outfit.py             # OutfitPlan schema
│   └── activity.py           # Activity constraints schema
├── memory/
│   └── user_memory.py        # UserMemory class (storage abstraction)
└── config/
    ├── loging_confi.py       # Structured logging setup
    ├── metrics.py            # MetricsCollector class
    └── alert.py              # AlertPolicyManager class

Appendix A: Coach Agent Code

# coach.py
from google.adk.agents import Agent
from ..tools.weather_tools import get_weather_smart
from ..tools.activity_tools import classify_activity
from ..tools.outfit_tools import plan_outfit
from ..tools.safety_tools import check_safety
from ..tools.memory_tools import get_user_preferences, update_user_preferences

coach_agent = Agent(
    name="coach_agent",
    model="gemini-2.0-flash-exp",
    instruction="""You are the Weather Outfit Coach...
    Workflow:
    1. Get user preferences to personalize response
    2. Extract city from query
    3. If activity mentioned, classify using classify_activity
    4. Get weather using get_weather_smart
    5. Plan outfit using plan_outfit
    6. Check safety using check_safety
    7. Combine into friendly, personalized response
    """,
    tools=[
        get_user_preferences, update_user_preferences,
        get_weather_smart, classify_activity,
        plan_outfit, check_safety
    ]
)

Appendix B: Weather Agent Code

# weather.py
from google.adk.agents import Agent
from ..tools.weather_tools import (
    get_current_weather,
    get_hourly_forecast,
    get_weather_smart
)

weather_agent = Agent(
    name="weather_agent",
    model="gemini-2.0-flash-exp",
    instruction="""You are a weather specialist agent.
    Your role:
    - Always call weather tools before providing info
    - Never guess or use training data
    - Return clean, structured forecast data
    - Focus only on weather, not clothing
    
    Rules:
    - Use get_weather_smart for efficiency (caches results)
    """,
    tools=[get_current_weather, get_hourly_forecast, get_weather_smart]
)

Appendix C: Stylist Agent Code

# stylist.py
from google.adk.agents import Agent
from ..tools.outfit_tools import plan_outfit

stylist_agent = Agent(
    name="stylist_agent",
    model="gemini-2.0-flash-exp",
    instruction="""You are a clothing and style advisor agent.
    Your role:
    - Take structured weather data and user preferences
    - Use plan_outfit tool to generate recommendations
    - Provide clear, practical advice with layers/accessories
    - Never call weather APIs yourself
    
    Style approaches based on persona:
    - Practical: Focus on function, comfort, simplicity
    - Fashion: Add style tips, color coordination, trends
    - Kid-friendly: Use fun language, prioritize safety
    """,
    tools=[plan_outfit]
)

Appendix D & E: Activity & Safety Agents

# activity.py
activity_agent = Agent(
    name="activity_agent",
    model="gemini-2.0-flash-exp",
    instruction="""Classify user activity into:
    - Work: Office, meetings (business_casual, low movement)
    - Sports: Hiking, biking (casual, high movement)
    - Formal: Dates, dinners (formal, low movement)
    - Casual: Walking, shopping (casual, medium movement)
    """,
    tools=[classify_activity]
)

# safety.py
safety_agent = Agent(
    name="safety_agent",
    model="gemini-2.0-flash-exp",
    instruction="""Review weather for risks.
    Safety thresholds:
    - Extreme cold: Below 20°F
    - Extreme heat: Above 95°F
    - Strong winds: Above 25 mph
    - Heavy rain/storms: Above 70% chance
    """,
    tools=[check_safety]
)

Appendix F: User Memory Implementation

# user_memory.py
from typing import Dict, Optional
from ..schemas.memory import UserPreferences, PersonaType, ComfortProfile

class UserMemory:
    """Manages long-term user preferences and profile data."""
    
    def __init__(self):
        self._memory_store: Dict[str, UserPreferences] = {}
    
    def get_preferences(self, user_id: str) -> UserPreferences:
        """Retrieve user preferences from memory."""
        if user_id not in self._memory_store:
            self._memory_store[user_id] = UserPreferences()
        return self._memory_store[user_id]
    
    def update_preferences(
        self, user_id: str,
        persona: Optional[PersonaType] = None,
        comfort_profile: Optional[ComfortProfile] = None,
        default_city: Optional[str] = None
    ) -> UserPreferences:
        current_prefs = self.get_preferences(user_id)
        if persona: current_prefs.persona = persona
        if comfort_profile: current_prefs.comfort_profile = comfort_profile
        if default_city: current_prefs.default_city = default_city
        return current_prefs

Appendix G: Metrics Implementation

# metrics.py
from contextlib import contextmanager
import time

class MetricsCollector:
    @contextmanager
    def measure_time(self, metric_name: str, labels=None):
        """Context manager to measure execution time"""
        start_time = time.time()
        try:
            yield
        finally:
            duration_ms = (time.time() - start_time) * 1000
            self.record_latency(metric_name, duration_ms, labels)
    
    def increment_counter(self, metric_name: str, value=1, labels=None):
        # Increments in-memory counter
        if self.enabled:
            self._write_custom_metric(...)
    
    def record_latency(self, metric_name: str, duration_ms: float, labels=None):
        # Records in-memory timer
        if self.enabled:
            self._write_custom_metric(...)

# Global instance
agent_metrics = MetricsCollector()

Appendix I: Structured Logging Configuration

# loging_confi.py
import logging, sys
from google.cloud import logging as cloud_logging

def setup_logging(service_name: str, level="INFO", 
                  enable_cloud_logging=True) -> logging.Logger:
    logger = logging.getLogger(service_name)
    logger.setLevel(getattr(logging, level.upper()))
    
    # Structured JSON format
    formatter = logging.Formatter(
        '{"time": "%(asctime)s", "level": "%(levelname)s", '
        '"service": "' + service_name + '", "message": "%(message)s"}',
        datefmt='%Y-%m-%dT%H:%M:%S'
    )
    
    console_handler = logging.StreamHandler(sys.stdout)
    console_handler.setFormatter(formatter)
    logger.addHandler(console_handler)
    
    # Cloud Logging integration
    if enable_cloud_logging:
        client = cloud_logging.Client(project=project_id)
        cloud_handler = client.get_default_handler()
        logger.addHandler(cloud_handler)
    
    return logger

Appendix J: Full System Test Log

✅ All tools imported successfully
[WARNING] No RAPIDAPI_KEY found - using mock weather data
✅ Weather: 65.0°F, partly cloudy
✅ Activity: sports, formality=casual
✅ Outfit: long-sleeve shirt or light sweater, no jacket
✅ Safety: high risk
✅ Memory: persona=practical, comfort=neutral
✅ Updated: persona=fashion, city=Seattle
✅ ADK Agent class imported
✅ All agents imported successfully
  - Coach has 6 tools
✅ Main app.py imported successfully
✅ ADK app name: app
✅ All schemas imported
✅ WeatherData: 65.0°F
✅ UserPreferences: practical
✅ UserMemory instance created
✅ Preferences stored
✅ Preferences retrieved: Portland
✅ Multiple users supported
✅ All 5/5 tests passed
✅ All tools functional
✅ All agents operational
✅ Main app ready
✅ Schemas validated
✅ Memory system integrated

Appendix L: Frontend README

Modern Chat Interface Features

Appendix P: Location-Aware Frontend Features

Outfit Icons Mapping

Appendix M: Three Deployment Methods

curl -X POST https://YOUR-ENDPOINT/api/chat \
  -H "Content-Type: application/json" \
  -d '{"message": "What should I wear in Seattle today?"}'

Appendix N: Advanced Alert Policies (MQL)

Problem with Simple Rate Alerts

Default alerts (e.g., > 5 errors/min) are rate-based. They don't distinguish between 5 errors in 10 requests (50% error rate) vs. 5 errors in 10,000 requests (0.05%).

Solution: Ratio-Based Alerts with MQL

-- MQL for Error Rate > 5%
fetch global
| { metric 'custom.googleapis.com/agent/agent_calls'
  | filter metric.status == 'error'
  | align rate(1m)
  | group_by [], [value_error: sum(value.agent_calls)]
  ;
  metric 'custom.googleapis.com/agent/agent_calls'
  | filter metric.status == 'success'
  | align rate(1m)
  | group_by [], [value_success: sum(value.agent_calls)]
  }
| join
| value [error_rate: cast_double(val(0)) / 
                    (cast_double(val(0)) + cast_double(val(1)))]
| condition error_rate > 0.05

Appendix O: Production Architecture (6 Services)

Tool Design Best Practices (Day 2)

Context Engineering Loop (Day 3)

The Fetch → Prepare → Invoke → Upload Pattern

Sessions vs. Memory: Complete Comparison

Sample Trace Walkthrough (Part 1)

User Query: "What should I wear for a walk at 7pm in Redmond?"

Step 1: User → Coach Agent
Input: "What should I wear for a walk at 7pm in Redmond?"
User ID: user_123 (no preferences stored yet)

Step 2: Coach → Memory Tool
Tool Call: get_preferences_for_user(user_123)
Response: {} (empty - no preferences)

Step 3: Coach → User (Followup)
"To personalize: Do you prefer practical, fashion, or kid-friendly style?"

Step 4: User → Coach
"Practical please"

Step 5: Coach → Memory Tool
Tool Call: set_preferences_for_user(user_123, "practical", "neutral")
Response: {persona: "practical", comfort_profile: "neutral"}

Sample Trace Walkthrough (Part 2)

Step 6: Coach → Activity Agent (A2A)
Message: {activity_text: "walk"}
Response: {category: "casual", formality: "casual", movement: "medium"}

Step 7: Coach → Weather Agent (A2A)
Message: {city: "Redmond", time_window: "evening"}
Response: {temp_c: 12.0, rain_chance: 0.6, wind_speed: 15.0, ...}
Source: cache (CACHE HIT - 125ms)

Step 8: Coach → Stylist Agent (A2A)
Message: {weather_context, activity, persona: "practical"}
Response: {top_outer: "light jacket", bottom: "jeans", ...}

Step 9: Coach → Safety Agent (A2A)
Message: {weather_context}
Response: {risk_level: "medium", safety_note: "High chance of rain..."}

Step 10: Coach → User
"For your evening walk in Redmond: Light jacket, jeans, waterproof 
shoes recommended. 60% chance of rain - bring an umbrella!"

Four Pillars of Agent Quality (Day 4)

Debugging: Resolving Test Failure

Error from Test Log

❌ App import failed: 'Flask' object has no attribute 'root_agent'

Root Cause

Test script `test_full_system_with_metrics.py` imported the Flask frontend `app.py` instead of the ADK backend `app.py`. Flask app has no `.root_agent` attribute.

The Fix

Multi-Framework & Multi-Service Patterns

Why A2A Enables Framework Agnosticism

Final System Status: Production Ready

The 5-Step Agentic Problem-Solving Process

From Concept to Execution

Level 4: The Self-Evolving System (Future)

Autonomous Creation & Adaptation

Level 4 agents can identify gaps in their capabilities and dynamically create new tools or agents to fill them.

Scenario: Solaris Headphones Launch

Project Manager Agent realizes it needs social media sentiment tracking,
but no such tool exists.

Step 1: Meta-Reasoning
Think: "I must track social media buzz for 'Solaris,' but I lack the capability."

Step 2: Autonomous Creation
Act: Invoke AgentCreator tool with mission:
  "Build agent that monitors social media for 'Solaris headphones',
   performs sentiment analysis, and reports daily summary."

Step 3: Observe & Deploy
Observe: New SentimentAnalysisAgent created, tested, and added to team.
Result: Agent dynamically expanded its own capabilities mid-task.

Tool Taxonomy: Four Primary Functions

Key Design Principles

Session vs Memory: The Workbench Analogy

"Outside-In" Evaluation Hierarchy (Day 4)

From Black Box to Glass Box

Trajectory Evaluation Steps

The Agent Quality Flywheel

Continuous Improvement Loop

LLM-as-a-Judge: Scalable Quality Evaluation

Why Traditional Metrics Fail for Agents

How do you measure the "accuracy" of a generated paragraph? Traditional ML metrics (precision, recall, F1) don't apply to open-ended agent outputs.

Multi-Agent Session Management (Day 3)

Production Session Considerations (Day 3)

From Prototype to Enterprise-Grade

References