Skip to content

Source Access and Export Setup

For the current in-app recovery controls, expanded provider coverage, and error reference, see Source Recovery and Diagnostics.

Aesop uses three collection methods, in descending order of reliability:

  1. Official account exports, retained unchanged and imported locally.
  2. Read-only snapshots of documented local application stores.
  3. Isolated browser profiles for services without a supported export API.

Never place browser cookies, OAuth credentials, API keys, .env.local, or application credential files inside data/ or imports/.

macOS Access

The collector needs read access to application data in your home directory.

  1. Install the native app with aesop service install.
  2. Launch a read-only discovery check through the app: open -n "$HOME/Applications/Aesop Agentic.app" --args discover.
  3. Review product-attributed macOS access prompts. For a source that reports Operation not permitted, review System Settings → Privacy & Security for Aesop Agentic. Grant only the source access required. If a protected application store requires Full Disk Access, enable it for the native app.
  4. A manual CLI run uses the launching terminal's permission context. Service access must be verified through the installed app; granting a generic Python runtime is not the recommended service setup.
  5. SQLite stores use online backup snapshots, including committed WAL changes. They do not require closing the source app. A snapshot timeout or failure is reported rather than falling back to an incomplete main database file. Close Electron apps when collecting their opaque LevelDB stores if a stable snapshot is needed; their coverage remains partial.

Legacy Python permissions are separate; do not reset shared permissions. Discovery and check confirm locations, not that every file can be read or that browser authentication remains valid. A real collection supplies that evidence.

Check access without modifying source stores:

aesop doctor --json
aesop collect --dry-run --json

ChatGPT

The official export is the authoritative readable source. OpenAI documents the current workflow at Exporting your ChatGPT history and data:

  1. Sign in to ChatGPT.
  2. Open Settings → Data controls → Export data.
  3. Confirm the export request.
  4. Download the ZIP from the email link before it expires.
  5. Put the ZIP in imports/chatgpt/, or import it directly:
aesop import chatgpt ~/Downloads/chatgpt-export.zip
aesop parse --source chatgpt_export

The desktop application’s .data files are still archived as raw evidence, but they are opaque and are not a substitute for the official export.

For incremental best-effort capture between exports:

aesop auth chatgpt
aesop collect --source chatgpt

The authentication window uses a dedicated profile under ~/.local/state/aesop-agentic/browser-profiles/chatgpt. Do not point it at your normal browser profile or copy cookies manually. Re-run aesop auth chatgpt if coverage reports blocked or the session expires.

Gemini

Google’s per-response export workflow is documented at Export responses from Gemini Apps. For bulk account activity, use Google Takeout:

  1. Deselect all products.
  2. Select My Activity and restrict activity data to Gemini Apps when that filter is available for the account.
  3. Create and download the export.
  4. Place the archive in imports/gemini_web/, or run:
aesop import gemini_web ~/Downloads/takeout.zip

Takeout/My Activity and individual response exports may expose different subsets, so retain both when possible. Gemini CLI sessions are collected separately from ~/.gemini/tmp/*/chats/; credential files such as oauth_creds.json and google_accounts.json are intentionally excluded.

For incremental browser capture:

aesop auth gemini_web
aesop collect --source gemini_web

Gemini Antigravity protobuf files are preserved but remain opaque until a stable decoder is available.

Replit

No documented, stable bulk export or API for Replit Agent conversation history was found during implementation. A project ZIP exports project files; it must not be treated as proof that Agent chats were exported.

Use all available layers:

  1. Close Replit before collecting its local Electron IndexedDB/LevelDB snapshot.
  2. Initialize the isolated browser profile:
aesop auth replit
aesop collect --source replit
  1. Export important Replit projects separately for source-code backup.
  2. For account-level portability, request a copy of personal data through Replit support or the privacy contact applicable to your account.
  3. Record the request date and place delivered archives in imports/replit/.

Replit browser and LevelDB coverage is best-effort and must remain partial or blocked when no conversation records can be decoded.

Local Coding Agents

  • Cursor: agent JSONL, workspace databases, the conversation search index, and detected Cursor CLI store.db files are collected. Cursor KV databases remain partially opaque.
  • Codex: sessions and archived sessions under ~/.codex/ are collected; auth.json is excluded.
  • Claude Code: project JSONL and session indexes under ~/.claude/projects/ are collected; .credentials.json is excluded.
  • Gemini CLI: project chat JSON is collected; OAuth/account files are excluded.
  • GitHub Copilot: VS Code chat-session JSON is readable. .xd stores are retained raw and may remain opaque. Close the editor before snapshotting.

Local Apps and Transcription Tools

  • Warp: the collector snapshots warp.sqlite through SQLite online backup. Agent conversations and AI query data are parsed when their schema is available.
  • Ollama: desktop chat tables and CLI prompt history are collected. Ollama API/CLI responses that no client persisted cannot be recovered retroactively.
  • LM Studio: saved .conversation.json files are collected. API-server traffic is not necessarily persisted by LM Studio.
  • superwhisper: recording metadata and its SQLite database are collected; model files are excluded.
  • MacWhisper: sessions, dictations, transcript text, and available AI summaries are read from its SQLite database; model files are excluded.

Routine Verification

After adding an export or refreshing authentication:

aesop collect
aesop parse
aesop coverage
aesop verify

Human-readable output appears under data/parsed/. Raw artifacts remain under data/raw/; parsing never modifies them. Treat partial, blocked, opaque, and empty as distinct states when assessing completeness.

Pi and Orca

Add "pi" and "orca" to [sources].enabled in existing configurations. New configurations include both. Keep "codex" enabled for Orca's managed Codex sessions and the underlying harness adapters enabled for other Orca agents.

Pi discovery reads only JSONL files below ~/.pi/agent/sessions/, or below $PI_CODING_AGENT_DIR/sessions when that environment variable is available to the collector. A LaunchAgent does not inherit your interactive shell settings. Use explicit paths for custom --session-dir locations and additional Orca profiles or installations:

[sources]
enabled = ["codex", "claude_code", "gemini_cli", "pi", "orca"]
pi_session_roots = ["~/Agent Sessions/Pi"]
orca_roots = ["~/Library/Application Support/Orca Development"]

These roots supplement the defaults. Use your complete enabled-source list when editing an existing configuration. Do not add another [sources] table.

Pi versions 1–3 are decoded as a complete persisted event log, with entry and parent links, alternate branches, model/provider metadata, tool calls/results, and compaction events retained. The document is not an active-branch replay. Unknown versions and malformed records are reported. Collection begins when a persisted session is available at one of the configured roots.

Orca discovery includes orchestration.db, allowlisted terminal-history metadata/output, and, under the codex source, codex-runtime-home/home/sessions/ and archived_sessions/. Mailbox messages are decoded only when the expected database schema is present. Terminal output is a flattened, possibly truncated record. Keys, cookies, account-session files, and browser profiles are not included by these new allowlists. The orchestration database and transcripts can still contain sensitive user content.

Remote Orca agents require a collector on the remote host or a deliberate export import; a local runtime's task metadata does not prove remote transcript coverage. Orca's custom harness homes also need explicit collection where they differ from the existing adapters. See the coverage matrix.

Resolving Current Collection Errors

  • iCloud and Superwhisper, Resource deadlock avoided: this is a filesystem read failure on a file iCloud has offloaded (a dataless placeholder), not a parser failure. macOS refuses the download to background processes and may block foreground ones.
  • Backups never read the iCloud copy. Aesop records what it last wrote to each destination in state_root/mirrors/<destination>.sqlite and checks files with metadata only, so evicted backup files stay in iCloud. A placeholder that cannot be confirmed from that record is overwritten, which needs no download.
  • Sources stored in iCloud (for example Superwhisper) must be local to be read. In Finder, choose Keep Downloaded for the source folder.
  • A file that cannot be read or written is reported and skipped; the rest of the mirror continues and the destination is marked partial. Up to 20 file errors are listed.
  • Each destination runs in a separate worker. It stops after stall_timeout_seconds (default 300) without progress or timeout_seconds (default 1,800) overall. Only whole-destination failures, such as an unmounted volume, are retried (at most twice); a stalled or partial mirror waits for the next run so a blocked read is not repeated.
  • Live SQLite indexes are copied as consistent snapshots; -wal and -shm files are not mirrored.
  • Folders in the destination that Aesop did not create (for example an older data/ copy) are reported as unmanaged content and never removed. Delete them yourself after review.
  • Warp, Operation not permitted: macOS privacy protection blocks files in other apps' containers. Grant Full Disk Access to Aesop Agentic in System Settings → Privacy & Security and retry through the installed app. A successful discovery check alone does not establish access to warp.sqlite.
  • ChatGPT/Replit, no fresh pages: run aesop login chatgpt or aesop login replit and complete sign-in in the dedicated window. If the service still has no conversation links, use official exports where available and retain the explicit coverage gap. Cached pages are not fresh collection.
  • Partial or opaque parsing: inspect data/parsed/manifest.json. failed and errors identify actual parser failures; partial and opaque record limits of the source format. Parser version 7 retries previously failed artifacts and reparses files when the decoder version changes.

Collection manifests under data/runs/ are the snapshot included in backups. Final operational summaries under ~/Library/Logs/Aesop Agentic/runs/ also include backup results. The dashboard uses the matching operational summary; those logs stay outside the mirrored tree so recording the result does not immediately invalidate the backup's checksums.