Time-Travel Debugging for AI Agents 🎬
Published: January 15, 2025
What if you could record an AI agent's session and replay it later? DevDuck's session recording feature enables exactly that - a time-travel debugger for AI agents.
The Vision
sequenceDiagram
participant User
participant DevDuck
participant Recorder
participant ZIP as session.zip
User->>DevDuck: session_recorder(action="start")
Recorder->>Recorder: Start capturing
User->>DevDuck: "analyze this code"
DevDuck->>Recorder: Record tool calls
DevDuck->>Recorder: Record agent messages
DevDuck->>Recorder: Record sys events
User->>DevDuck: session_recorder(action="snapshot")
Recorder->>Recorder: Capture state
User->>DevDuck: session_recorder(action="stop")
Recorder->>ZIP: Export all events
Note over ZIP: Complete session<br/>ready for replayThree Layers of Recording
flowchart TB
subgraph "Agent Layer"
AM[Agent Messages]
AD[Agent Decisions]
AS[Agent State]
end
subgraph "Tool Layer"
TC[Tool Calls]
TR[Tool Results]
TD[Tool Duration]
end
subgraph "System Layer"
FO[File Operations]
NO[Network Ops]
EV[Environment]
end
AM & AD & AS --> Buffer
TC & TR & TD --> Buffer
FO & NO & EV --> Buffer
Buffer --> Export[session.zip]
style Buffer fill:#f9f,stroke:#333,stroke-width:4pxEvent Types
@dataclass
class RecordedEvent:
timestamp_ns: int # Nanosecond precision
layer: str # "sys", "tool", "agent"
event_type: str # "file.open", "tool.call", "message"
data: Dict[str, Any] # Event-specific data
trace_id: str # For correlation
Recording a Session
Start Recording
🦆 session_recorder(action="start")
# Output:
🎬 Recording started: session-20260203-143052
Events will be captured at sys/tool/agent layers.
Create Snapshots
# After important milestones
🦆 session_recorder(action="snapshot", description="after API integration")
# Output:
🎬 Snapshot #1 created: after API integration
Stop and Export
🦆 session_recorder(action="stop")
# Output:
🎬 Recording stopped and exported!
Session: session-20260203-143052
Events: 847
Snapshots: 3
Export: /tmp/devduck/recordings/session-20260203-143052.zip
The Session ZIP Format
session-20260203-143052.zip
├── events.jsonl # All events in JSON Lines format
├── snapshots.json # State snapshots with conversation history
├── metadata.json # Session info
└── session.pkl # Serialized state (for true replay)
events.jsonl
{"timestamp_ns": 1706973052000000000, "layer": "agent", "event_type": "session.start", "data": {"session_id": "session-20260203-143052"}}
{"timestamp_ns": 1706973053000000000, "layer": "tool", "event_type": "tool.call", "data": {"name": "shell", "args": {"command": "ls -la"}}}
{"timestamp_ns": 1706973054000000000, "layer": "sys", "event_type": "file.open", "data": {"path": "/tmp/test.txt", "mode": "r"}}
snapshots.json
[
{
"snapshot_id": 1,
"timestamp": 1706973100.5,
"agent_messages_count": 5,
"agent_messages": [
{"role": "user", "content": "analyze this code"},
{"role": "assistant", "content": "I'll examine..."}
],
"system_prompt": "You are DevDuck...",
"last_query": "analyze this code",
"last_result": "Found 3 issues..."
}
]
Loading and Analyzing Sessions
from devduck import load_session
# Load a recorded session
session = load_session("~/Desktop/session-20260203-143052.zip")
print(session)
# LoadedSession(id=session-20260203-143052, events=847, snapshots=3, duration=245.3s)
# Filter events by layer
tool_events = session.get_events_by_layer("tool")
print(f"Tool calls: {len(tool_events)}")
# Filter by event type
file_opens = session.get_events_by_type("file.open")
print(f"Files opened: {len(file_opens)}")
# Get events up to a snapshot
events_before_snapshot = session.get_events_until_snapshot(2)
Resuming from Snapshots
sequenceDiagram
participant User
participant Session as LoadedSession
participant Agent as DevDuck Agent
User->>Session: resume_from_snapshot(2)
Session->>Session: Load snapshot #2
Session->>Agent: Restore messages
Session->>Agent: Restore state
Session->>User: Resume result
User->>Agent: Continue working...
Note over Agent: Conversation<br/>history intact!Resume API
from devduck import load_session
session = load_session("session.zip")
# Resume from snapshot (restores agent.messages!)
result = session.resume_from_snapshot(2, agent=my_agent)
print(f"Restored {result['messages_restored']} messages")
# Resume and continue with new query
result = session.resume_and_continue(
snapshot_id=2,
new_query="what was I working on?",
agent=my_agent
)
print(result['agent_result'])
CLI Resume
# Resume from latest snapshot
$ devduck --resume ~/Desktop/session-20260203-143052.zip
# Resume from specific snapshot
$ devduck --resume session.zip --snapshot 2
# Resume and run query
$ devduck --resume session.zip "continue where we left off"
Use Cases
1. Debugging Agent Behavior
flowchart LR
Bug[Bug Report] --> Load[Load Session]
Load --> Analyze[Analyze Events]
Analyze --> Find[Find Root Cause]
Find --> Fix[Fix Agent]session = load_session("problematic-session.zip")
# Find the failing tool call
for event in session.get_events_by_layer("tool"):
if "error" in str(event.data).lower():
print(f"Error at {event.timestamp_ns}: {event.data}")
2. Training Data Generation
# Extract all successful interactions
successful_sessions = []
for session_path in Path(RECORDING_DIR).glob("*.zip"):
session = load_session(str(session_path))
if session.metadata.get("status") == "success":
successful_sessions.append(session.to_dict())
# Export for training
with open("training_data.json", "w") as f:
json.dump(successful_sessions, f)
3. Long-Running Task Continuation
# Start a long analysis session
🦆 session_recorder(action="start")
🦆 analyze all Python files in this project
# ... computer crashes or you close terminal ...
# Later, resume where you left off
$ devduck --resume /tmp/devduck/recordings/session-*.zip
✅ Resumed from snapshot #3
Messages restored: 12
Working directory: /Users/dev/project
4. Collaboration & Sharing
# Share your agent session with a colleague
$ cp /tmp/devduck/recordings/session-*.zip ~/Desktop/agent-session.zip
$ # Send to colleague...
# Colleague loads and continues
$ devduck --resume agent-session.zip "what did you find?"
Recording Architecture
flowchart TB
subgraph "Recording System"
Recorder[SessionRecorder]
Buffer[EventBuffer]
Hooks[System Hooks]
end
subgraph "Hooks"
OpenHook[builtins.open]
RequestsHook[requests.get]
end
subgraph "Events"
ToolCall[Tool Calls]
AgentMsg[Agent Messages]
SysEvents[System Events]
end
OpenHook --> SysEvents
RequestsHook --> SysEvents
ToolCall --> Buffer
AgentMsg --> Buffer
SysEvents --> Buffer
Buffer --> Recorder
Recorder --> Export[ZIP Export]
style Recorder fill:#f9f,stroke:#333,stroke-width:4pxEvent Buffer
class EventBuffer:
"""Ring buffer for recording events across all layers."""
def __init__(self, max_events: int = 10000):
self.events = deque(maxlen=max_events)
def record(self, layer: str, event_type: str, data: dict):
event = RecordedEvent(
timestamp_ns=time.time_ns(),
layer=layer,
event_type=event_type,
data=data,
)
self.events.append(event)
System Hooks
def _install_hooks(self):
"""Install hooks to capture OS-level events."""
# Hook: builtins.open
self._original_open = builtins.open
def traced_open(file, mode="r", *args, **kwargs):
self.record_sys_event("file.open", {"path": str(file), "mode": mode})
return self._original_open(file, mode, *args, **kwargs)
builtins.open = traced_open
# Hook: requests.get
import requests
self._original_requests_get = requests.get
def traced_get(url, *args, **kwargs):
self.record_sys_event("http.get", {"url": str(url)[:200]})
return self._original_requests_get(url, *args, **kwargs)
requests.get = traced_get
Best Practices
Recording Tips
- Start early: Begin recording at session start
- Snapshot often: After major milestones
- Describe snapshots: Use meaningful descriptions
- Clean up: List and delete old sessions periodically
Storage Considerations
- Events are truncated to prevent huge files
- Ring buffer limits to 10,000 events
- Sensitive data (API keys) is redacted
- ZIP compression reduces file size
Future: Replay Mode
sequenceDiagram
participant User
participant Replayer
participant Events
participant Agent
User->>Replayer: replay_events(callback=handler)
loop For each event
Replayer->>Events: Get next event
Events->>Replayer: Event data
Replayer->>Agent: Apply event
Agent->>User: Show result
User->>Replayer: Continue/Pause
endComing soon: Step-by-step replay with pause/resume, allowing you to "debug" an agent session like debugging code.
Record. Analyze. Resume. Repeat. 🎬