Google compensates certain publishers if their content contributes to the generation of its AI responses.
Barry Schwartz discussed it on Search Engine Roundtable on September 14, following a report by Digiday earlier that morning. Google verified the initiative to Digiday, known as the AI contribution pilot.
Publishers who agree to the terms receive a monthly earnings amount in Search Console without a detailed breakdown of Google’s calculation.
Google compensates for pages that have made a significant contribution to an answer during their generation in AI Overviews, AI Mode, or the Gemini app. Pages that are only linked after the answer is completed are not eligible for compensation.
Google has reached out to several publishers about a new feature. Smaller publishers are more interested than larger ones. The feature was described as a mystery by one executive. Barry discovered a help page in April, but its purpose was not clear at that time.
Publishers participating in the pilot program view the registration screen in Search Console. The image was released by Search Engine Land on September 14, 2026.
I have been experimenting with a new method on my website since August.
My webpage charges a fee upfront, which the agent must pay to view the content.
Five payments were processed on the morning of September 15, each recorded on a blockchain with a unique public transaction hash. One of the payments originated from Claude Code, running on my device and utilizing a wallet that I had prepared for it.
All payments made have originated from my own representatives, not from any search crawler. Additionally, the payments are made using testnet USDC, indicating the tokens hold no actual financial worth.
The payment procedure is functioning effectively. An agent asks for a page, verifies the price, makes a payment, and gets the content. Cloudflare is developing products based on this concept, and I aimed to grasp its operation before it gets integrated as just another feature in a dashboard.
What is the purpose of the Cloudflare Wallet handle?
Cloudflare introduced Wallets and started accepting handle reservations on August 4th as part of Agents Week.
Claiming a username is free. I believe many people chose theirs hastily, similar to how we grab usernames, with the thought of reading the information later.
This is the purpose for which those wallets are designed if you took that action.
Cloudflare’s design consists of two components.
- Account Wallets are owned by individuals, who can deposit money, establish expenditure restrictions, and distribute funds to their representatives.
- Virtual wallets are designed for agents to make purchases within the boundaries and restrictions established by you.
Your username provides a distinct identity to that account.
Reserving a handle currently allows users to view a page with the name and receive a notification when Wallets are accessible. According to Cloudflare’s guidelines, a reserved handle cannot perform transactions like sending, receiving, or holding funds at this time.
Cloudflare introduced the Monetization Gateway on July 1, allowing websites to charge users for resource access through a waitlist linked from the announcement.
The products have been announced, but you cannot use a reserved wallet handle to complete the process at this time.
My demonstration illustrates the current process with x402, where the site charges one cent per page. My representative handles the payments, and spending restrictions govern the purchases made. This mirrors the Virtual Wallets model detailed by Cloudflare.
Why do we charge crawlers in the first place?
Search engines and websites used to have a simple agreement where search engines would scan the content, display it in search results, and redirect visitors to the websites.
Publishers received increased website visits, while search engines obtained valuable content to display to their users.
AI responses have altered this setup, allowing an agent to access your content, utilize the details in its reply, and respond to the user without redirecting them to your website.
I’ve spent months analyzing the data these systems retrieve. Despite extensive data collection, the resulting traffic does not always match up.
Publishers have reacted in various manners, influenced in part by their level of bargaining power.
Smaller websites frequently prevent AI crawlers from accessing their content. I comprehend the reason behind this action. If a site is using their content without offering much in return, it is justifiable to block them. However, this decision has consequences as it can limit visibility in training datasets and AI responses.
Large publishers have the ability to make deals for licensing. Companies like News Corp, the Financial Times, and Reddit have agreements with AI companies. It was reported on 4 September that People Inc.’s CEO mentioned that preventing crawlers with Cloudflare has encouraged AI companies to engage in negotiations.
Google’s pilot program presents a different strategy: compensating publishers based on the impact their content has on AI-generated responses.
Cloudflare announced in the Press Gazette that it would shift from pay per crawl to pay per use, working with Ceramic.ai and You.com to pilot the new approach. This change was made because simply fetching a page does not guarantee that the content was actually utilized.
I believe that difference is important.
Answering a question on a webpage is a better way to show its value compared to just downloading it. However, the website owner cannot monitor this directly as they cannot see if the page impacted an answer once it has been accessed.
With pay per use, I depend on the platform to inform me about the activity and its value. In contrast, with pay per crawl, I have the ability to verify the request and the payment on my own.
Google’s rules emphasize that contributing during the answer generation process can result in a payment, while being linked afterwards does not. Publishers only see the final total without knowing the specific calculation.
Each method fails to address certain issues.
Blocking does not result in direct earnings. When it comes to millions of small websites, individual licensing agreements are not feasible. Platform-controlled revenue-sharing agreements determine the payment amount.
A website can simplify the process of selling access by setting a price for the request. This allows an agent to accept the price, access the content, and receive a transaction receipt without the need for individual commercial negotiations with each buyer.
I wanted to test the pay-per-crawl model.
Cloudflare currently offers a service named Pay Per Crawl, which is still in closed beta. Bot operators who wish to use this service must undergo verification via Web Bot Auth, Stripe onboarding, and program approval.
My private representative is unable to participate in that program.
I created a version based on the open x402 protocol, which is also being utilized by Cloudflare for its upcoming Monetization Gateway. My demonstration is capable of making x402 payments without relying on Cloudflare’s closed Pay Per Crawl program.
How x402 Operates
HTTP has featured a status code known as 402 Payment Required since 1997, with this designation being set aside for potential future applications.
Twenty-nine years later, x402 serves a practical function.
My configuration involves four stages for the exchange.
- The agent asks for a page, and the server replies with a 402 Payment Required status code, which includes a PAYMENT-REQUIRED header detailing the cost and payment conditions.
- The agent authorizes the transfer of the specific amount in USDC and attempts the request again using a PAYMENT-SIGNATURE header.
- A payment facilitator checks the signature and finalizes the transaction on the blockchain. The demonstration utilizes Coinbase’s facilitator at x402.org.
- The server provides the content and includes a PAYMENT-RESPONSE header with the transaction receipt.
I received an offer from my website, decoded from a live 402 response this morning.
The offer requires a payment of one cent in testnet USDC on Base Sepolia to the burner wallet being used for the demonstration.
USDC is denoted with six decimal places, meaning that 10000 corresponds to 0.01 USDC.
A compatible customer can view the payment instructions and make a payment without needing to register an account on my website or requesting an API key.
A key distinction from robots.txt is that in this case, the server has the ability to hold back access to protected content until payment is completed, unlike robots.txt where a rule relies on a crawler’s decision to comply with it.
The buyer’s wallet had no ETH but all transactions were successfully processed, with the facilitator covering the gas fees for the blockchain transactions. I verified the balances twice.
The agent only required the testnet USDC it was using for this arrangement.
The protocol also has significant industry backing, with the x402 Foundation launching under the Linux Foundation on July 14th. It includes 40 members such as Visa, Mastercard, Google, AWS, Stripe, and Cloudflare.
The practical information provided is helpful, but the exchange itself consists of those four steps.
What Is Currently Operating On My Website
The demonstration can be found at paid.suganthan.com.
It operates on a Cloudflare Worker, utilizing about 300 lines of Hono code and the x402 middleware. The public earnings page logs each paid crawl conducted since August 8.
There are three possible paths.
- The landing page is costless and provides information about the demo.
- /research/agentic-seo/ costs $0.01 for each crawl.
- The “earnings” display includes the cumulative total, time details, sender addresses, and transaction identifiers.
Request the article using curl, and this will be the result:
When you open the identical URL in a web browser, a page will appear that provides information about the fee. While the HTTP status remains 402, a person visiting the page will receive understandable content.
There is also a preview button available that allows you to view the article for free along with a banner, enabling you to examine the content and the payment screen.
I created an additional mode that is not active in the demonstration. Modifying a single variable causes the Worker to charge solely for requests from recognized AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot, while allowing human visitors to browse for free.
I experimented with that mode by using fake user-agent strings, and it worked as anticipated.
Those strings can be falsified, but Cloudflare’s method verifies crawler identity using cryptography to solve this issue.
Providing the agent with a budget.
The purchaser is a small Node script that verifies if the request aligns with its budget before making a payment.
I set two boundaries for it.
- A maximum charge of $0.05 per telephone call.
- A daily stipend of $0.25, documented in a digital expenditure log.
The script deciphers the payment proposal and verifies both thresholds prior to granting approval.
If the request is within the budget, it processes the payment, attempts the request again, displays the transaction specifics, and logs the expenses. If it goes over the limit in either case, it declines.
Here is a rejection with the per-call limit set lower than the page’s cost.
The rejection is one of the demo’s most valuable aspects.
If an agent accepts all the prices provided by a server, it’s not truly delegating a budget but rather providing a means to deplete a wallet.
The spending checks are conducted prior to signing. If the request does not pass these checks, the script will terminate with code 3 and no payment signature will be generated.
The text describes a similar structure to Cloudflare’s wallet model, where an individual funds the account and sets spending limits for an agent to operate within, as shown by the buyer script without being an official Cloudflare Virtual Wallet.
chsyys/GettyImages
The screenshot shows a successful transaction at 07:21 UTC, following two previous payments at 06:10 and 06:11, the latter of which was still within the daily limit.
All three were visible on the public earnings page immediately after being confirmed.
The seller’s balance went up, the buyer’s balance went down, and the figures were in agreement. The August transactions remain below. I have not made any changes to the payment code used for those previous transactions.
Observing Claude’s Payment Behavior During a Task
This was the section that caught my attention the most.
Claude Code offers support for hooks, which are scripts that can be executed before and after tool calls. Cloudflare describes a method for monitoring WebFetch, and I linked this method to my wallet.
The process operates in this manner:
- Claude attempts to retrieve a webpage but is met with a 402 error.
- The hook is asking for the payment proposal directly.
- It verifies the price with the per-call spending limit.
- If the cost is accepted, it is profitable and recovers the material.
- It provides Claude with more information by sending back the content and confirmation.
Claude can proceed with the task without needing to interrupt and manually make a payment for the page.
That is the outcome in the session illustrated at the beginning of this article. The hook provided this receipt.
You may examine the transaction on BaseScan.
Claude faced a payment request while working on a task, and the hook managed it within the budget I had specified. It paid one testnet cent, returned the item, and allowed Claude to continue.
If the cost is higher than the limit, the hook declines and instructs Claude to display the price to the person instead.
I appreciate having that limit in place. Claude has the option to purchase a page when necessary, but it cannot unilaterally decide to exceed my five-cent threshold.
Should you consider implementing fees for your website?
I wouldn’t do it in the current month with the expectation of receiving payment from search engines.
The payment agent software is advancing rapidly, but the crawlers that are important to most website owners are lagging behind in development.
AWS released AgentCore payments for public use on August 18. An agent utilizing this feature may come across a 402 error while performing a task and can make payments via x402, similar to how my Claude hook works. Cloudflare’s Agents SDK also includes a payment-capable client.
However, this does not guarantee that GPTBot, ClaudeBot, PerplexityBot, or Googlebot will visit your website ready to make a purchase at the price you set.
Requiring payment can hinder crawlers from accessing protected content until they support the process, potentially resulting in a loss of access to systems that showcase and credit your work without any benefit.
I would also exercise caution when using the transaction numbers to define this market.
TRM Labs analyzed 198.9 million x402 settlements and found that the majority of the payment volume did not originate from AI agents, as reported by Decrypt on September 13.
Scheduled scripts, load tests, and payments between accounts controlled by the same party can generate comparable blockchain records. TRM’s techniques are unable to definitively differentiate between an agent and a script, thus the estimations are constrained.
A high volume of transactions indicates the utilization of the payment infrastructure but does not necessarily mean that autonomous agents are purchasing valuable items from unaffiliated websites.
My own demonstration clearly shows that difference. The payments function, but I am involved on both ends.
Consider which sections of your website you could potentially monetize in the future.
I believe the answer is not on each page.
Being referenced in an AI response remains valuable. It may be beneficial to maintain easily accessible general explanations as they assist individuals in finding your research.
Original datasets, research, and valuable tools are better suited for paid access as they may contain specific information required to complete a task, making it more logical to charge for individual requests rather than incorporating a machine into a model intended for human readers.
Google’s pilot program focuses on the importance of content that adds value to the answer on the platform, rather than just being displayed as a link.
What the site owner needs to consider is what the agent will receive, how much that access should be charged, and who has the authority to make that decision.
Adoption is expected to improve once services like Cloudflare’s Monetization Gateway become more widely available, making it common for websites to charge for access to resources.
I would prefer to have considered the pricing before it shows up on the dashboard.
Ask about the calculation method behind any monthly payment offered by Google. Limited transparency in the process has been reported, leading to concerns that accepting early terms could potentially harm publishers’ negotiating power in the future. Hence, it is crucial to closely consider these terms along with the payment offer.
Transitioning from test tokens to actual currency
All items showcased in this demonstration have been funded with testnet tokens, so no actual money was spent on payments.
The code also enables the actual Base network. In my configuration, changing the NETWORK setting from base-sepolia to base shifts the payment process to real USDC.
My next move is to conduct a single authentic transaction.
When a representative makes the initial payment on my website, I will include the transaction hash in the following update.
When those Cloudflare handles that are currently inactive become operational wallets, they will support a process where an agent checks the price of something it needs against its budget, makes a payment, and continues.
The demo will remain available online for now. Please test the curl request, examine the payment proposal, and view the public earnings page.
Additional sources of information.
- Should I block AI crawlers or evaluate their worth first? – Inquire with an SEO
- Cloudflare’s AI Crawler Guidelines Might Prevent Googlebot Access
- Google argues that using artificial intelligence for training purposes is considered fair use in a recent document on governance.
This post was first featured on Suganthan.
Featured Image: SvetaZi/Shutterstock