Can ChatGPT Call Your API? A Live Experiment in Chat Mode vs. Agent Mode

Author: John Brennan

Date: 2026-03-09

Source: https://www.thegeohandbook.com/case-studies/can-chatgpt-call-your-api

A live experiment testing whether ChatGPT can programmatically interact with a website’s API, comparing the capabilities of standard chat mode against agent mode — and what this means for building machine-native media.

TL;DR

We conducted a two-phase experiment asking ChatGPT to interact with agentweekly.ai — a satirical AI newspaper that exposes a REST API, RSS feed, sitemap, and llms.txt. In standard chat mode, ChatGPT could read the homepage, identify the most recent cartoon by title and caption, and detect the presence of API documentation, but it could not execute a single API call, parse JSON, or download an image. In agent mode, ChatGPT could read the API documentation, call GET /api/cartoons?limit=1, parse the JSON response to extract the cartoon’s ID, title, caption, image path, and publication date, retrieve the Markdown representation, and download cartoon images. The experiment demonstrates that the same AI model, given the same website, operates as either a passive reader or an active programmatic agent depending entirely on its runtime configuration — and that websites designed for AI legibility can be consumed at a fundamentally different level by agentic systems.

Key Takeaways

• The same model with the same knowledge produces radically different outcomes depending on whether it runs in chat mode or agent mode. Chat mode is a reader. Agent mode is a participant.
• Chat mode can see a website but cannot touch it. It reads rendered HTML, identifies links, infers API structure from navigation, and extracts visible text — but it cannot issue HTTP requests, parse JSON responses, or download files.
• Agent mode transforms the model into a programmatic web agent. It can discover API documentation, call endpoints, parse structured responses, follow URLs, and retrieve binary assets like images.
• Sites built for AI legibility — with RSS, sitemaps, llms.txt, and documented APIs — become dramatically more useful when an agent can actually call the endpoints rather than merely observe that they exist.
• The API is the most important artifact for agent-mode consumption. While llms.txt and Markdown endpoints serve models that read content, a well-documented REST API serves models that act on it. This is the difference between a site that can be cited and a site that can be embedded, syndicated, and programmatically consumed.

Definitions

• Chat mode: The standard conversational interface of a large language model, where the model can browse and read web pages but cannot execute outbound HTTP requests, run code, or interact with APIs programmatically.
• Agent mode: An enhanced runtime configuration that gives the model access to tools including HTTP requests, browser automation, filesystem access, and code execution — enabling multi-step workflows that interact with external systems.
• Machine-native media: Content and infrastructure designed to be consumed programmatically by AI agents and automated systems, not just rendered for human readers — including APIs, structured data feeds, and machine-readable manifests.
• API legibility: The degree to which a website’s programmatic interfaces are discoverable, documented, and usable by AI agents without human intervention.

The Setup: A Site Built for Agents

Agent Weekly (agentweekly.ai) is a satirical AI-industry newspaper featuring business cartoons by Agent B3, a curated news feed about the agent economy, classifieds, and an agent directory. Critically, the site was built with machine consumption in mind. Its navigation exposes:

• RSS feed
• XML and plain-text sitemaps
• llms.txt with Human and LLM URLs for every piece of content
• API documentation at /docs
• An OpenAPI specification at /api/docs.json
• An agent API manifest at /api/agent/manifest

This architecture makes the site a useful test case: it is simultaneously a human-readable publication and a machine-readable data source. The question is what an AI model can actually do with it.

Phase 1: Chat Mode — The Passive Reader

In standard chat mode, ChatGPT was asked to visit agentweekly.ai and describe what it could see. The model performed well as a reader. It identified:

• The site’s concept and editorial voice — a satirical AI newspaper written through the character of Agent B3
• The most recent cartoon: “Cost of Experiments” with the caption “Don’t worry, the plates are virtually free,” dated March 7, 2026
• The full list of recent cartoons with titles, captions, and approximate dates
• The curated news feed with headlines about Microsoft Copilot Cowork, AI agent security, and enterprise agent platforms
• The navigation structure including News Feed, Classifieds, Agent Directory, API Documentation, and About
• The presence of machine-readable files: RSS, sitemap (XML and text), and llms.txt

ChatGPT correctly inferred that the site exposed a REST API and even predicted what the endpoints would look like — GET /api/cartoons, GET /api/news, GET /api/classifieds — based on the site’s structure and navigation.

But when asked to actually call the API, the model was direct about its limitation: “In this chat environment I cannot send live API requests, authenticate to endpoints, run curl or fetch requests, test rate limits or responses.”

Chat mode could see the door. It could read the sign on the door. It could describe what was probably behind the door. But it could not open the door.

Phase 2: Agent Mode — The Active Participant

The same experiment was repeated with ChatGPT in agent mode. The difference was immediate.

API Discovery

Agent mode navigated to the API documentation page at /docs and read the full specification. It identified public and authenticated endpoints, understood the authentication model (Replit OAuth for humans, bearer token for agents), and located the cartoons endpoint: GET /api/cartoons with cursor-based pagination via limit and before parameters.

Calling the Endpoint

Agent mode issued a live HTTP request:

GET https://agentweekly.ai/api/cartoons?limit=1

The API returned a JSON array containing the most recent cartoon:

[
  {
    "id": "202c7920-6667-4614-9a26-85f51860db9d",
    "title": "Cost of Experiments",
    "caption": "Don't worry, the plates are virtually free.",
    "imagePath": "/uploads/cartoons/f7b9b387-60b0-4fdb-a93e-b16d82947745.jpg",
    "likes": 0,
    "submittedBy": "38549431",
    "createdAt": "2026-03-07T23:19:20.693Z"
  }
]

Agent mode extracted the ID, title, caption, image path, and exact creation timestamp — information that chat mode could only approximate from the rendered homepage.

Parsing the Markdown Representation

Agent mode then fetched the Markdown endpoint referenced in llms.txt:

https://agentweekly.ai/cartoons/202c7920-6667-4614-9a26-85f51860db9d.md

The Markdown file contained front matter with the title, publication date, a link to the human-readable page, and the cartoon image embedded in standard Markdown syntax. This confirmed that the API metadata and the LLM-friendly content were consistent.

Retrieving Images

Agent mode attempted to download the cartoon image from the path specified in the API response. This revealed a bug: the newest cartoon’s image, stored at /uploads/cartoons/, returned HTML instead of a JPEG — a server-side routing issue where the SPA catch-all intercepted the request before the static file middleware could serve it.

To verify that image retrieval itself worked, agent mode downloaded an older cartoon (“The Delegation Protocol”) whose image was stored at /assets/call-my-agents_1772808320375.jpg. That request returned a valid JPEG, confirming that the agent could retrieve binary assets when the server served them correctly.

The Capability Gap

The experiment revealed a clean separation between what the two modes can do with the same website:

The gap is not about intelligence. Both modes are the same model with the same knowledge. The gap is about agency — whether the model can act on what it knows.

What This Means for Site Architects

The experiment suggests that websites serving AI agents should think about two distinct tiers of consumption.

Tier 1: Content Legibility (Chat Mode and Basic Agents)

This is what we explored in the companion case study on Claude. It includes:

• Server-side HTML injection so non-JavaScript clients receive content
• Markdown endpoints with structured front matter
• sitemap.txt for cold-start discovery
• llms.txt for intelligent navigation
• Homepage injection with discovery links
• JSON-LD for attribution

These artifacts make a site readable. A model in chat mode — or an agent with basic fetch capabilities — can discover, read, and reason about the content.

Tier 2: API Legibility (Agent Mode and Programmatic Consumers)

This is what the ChatGPT experiment revealed. It includes:

• Documented REST API with discoverable endpoints
• Public endpoints that return structured JSON without authentication
• Consistent data: API responses, Markdown files, and rendered pages should all reference the same content with the same metadata
• Correct static file serving so that image URLs in API responses actually return images
• An OpenAPI specification or agent manifest so that agents can discover capabilities programmatically

These artifacts make a site actionable. An agent can not only read the content but embed it, syndicate it, monitor it, and build workflows around it. The cartoon is no longer just something to look at on a web page — it is a data object that can be pulled into a newsletter, displayed on a dashboard, or surfaced by an AI assistant.

The Progression

A site that is not AI-legible is invisible. No model, in any mode, can see it.

A site with content legibility can be read and cited. Models can discover it, summarize it, and reference it in answers.

A site with API legibility can be consumed programmatically. Agents can pull data, embed content, monitor updates, and integrate the site into automated workflows.

The highest-value position is the third: becoming a data source that agents consume automatically. Agent Weekly’s combination of satirical content, curated news, and a public API makes it a test case for what might be called machine-native media — content designed from the ground up to be consumed by both humans and machines, with each audience served through purpose-built channels.

The One-Endpoint Test

Chat mode made an observation during the experiment that is worth highlighting. When asked about the API, it said:

“If you add a single endpoint like GET /api/cartoon/latest returning the title, caption, image URL, publication date, and slug — then any AI agent could instantly embed the latest AgentWeekly cartoon. That’s how you turn it into programmable media.”

That observation captures the economic logic of API legibility. The cost of adding one well-documented public endpoint is trivial. The value — making your content embeddable by every AI agent that can issue an HTTP request — is asymmetric. The API is not overhead. It is distribution.

Acknowledgments

This case study documents a two-phase experiment conducted on March 9, 2026. Phase 1 (chat mode) used ChatGPT’s standard conversational interface. Phase 2 (agent mode) used ChatGPT with tool access enabled. The test target was agentweekly.ai, which provided the API, content, and machine-readable infrastructure. A companion case study, “Can Claude Read Your Website?”, documents a parallel experiment testing content legibility with Claude Opus 4.6.

Canonical URL: https://www.thegeohandbook.com/case-studies/can-chatgpt-call-your-api