Decentralised News · Agentic Finance · Decision research
By Heath Muchena · Source review 2026-11-13 · Instrument version 1.0.0
What Matters
- A payment challenge creates an offer; it does not establish paying demand.
- AWS WAF and Cloudflare document different rollout, charging and rule-precedence boundaries. Match the policy to the actual implementation.
- DN’s local record flags contradictions and missing evidence in your selections. It does not inspect a website, authenticate crawlers, verify a deployment or forecast revenue.
DN Alpha Thesis
A publisher considering paid AI access has a more useful first question than “What should each crawl cost?” It is: which requests should receive an offer, under which terms, through a route the intended client can use?
A working payment mechanism can coexist with an ineffective business policy. An earlier security rule may block the buyer. The price may apply to a different resource than expected. The terms may arrive too late. A successful purchase may generate more support work than the publisher anticipated.
This guide compares documented AWS WAF and native Cloudflare Pay Per Crawl boundaries, then gives publishers a small decision record. The original contribution is the policy review: preserve conflicts and unknowns before they disappear behind a single “monetization enabled” label.
DN Evidence Block
Source review: 13 November 2026. Ten first-party documentation pages establish described capabilities, access status, charging boundaries and configuration constraints. Provider documentation is not independent evidence of actual traffic, customer identity, performance or revenue. DN did not configure either service, make a payment, inspect a customer account or run a crawler experiment. The tool examples are synthetic. The record is a proposed editorial method, not an automatic feed or deployed paywall.
Original DN instrument · version 1.0.0
DN Crawl Access Policy Record
Review one declared access policy. The local record flags selected conflicts and unresolved evidence; it does not inspect a website, account, crawler or deployment.
Enter your assumptions or load a labelled example. Required blank numbers stay missing; unknown evidence remains unknown. Use nonpersonal labels only. This instrument makes no network requests and uses no persistent browser storage.
Use your own inputs
No current result. Enter inputs or load an illustrative example.
Self-reported categorical inputs only. No live request, configuration validation, identity verification, legal decision, revenue estimate or deployment action.
Define the resource and the business purpose
Start with one resource class and one requested use: a public article for search discovery, a premium dataset for retrieval, or a licensed archive for another purpose. Record the intended audience, the content version and the owner of the access decision. “AI traffic” is too broad to define either a product or a buyer.
An editorial publisher may want a free summary discovered while charging for a deeper research resource. Another publisher may prioritise a negotiated licence. Both can be coherent policies. The important question is whether the proposed charging boundary supports the business objective and can be explained to the people responsible for security, publishing and accounts.
Keep the discovery path inspectable. Cloudflare’s advanced configuration documents URI-based exclusions and permanently free discovery/security paths. That gives a concrete way to distinguish a paid section from navigation; it does not prove that a particular crawler will discover, understand or purchase it. [8]
Write the counterfactual as a question: what would this same resource and audience receive under the current policy? A change in request volume is interpretable only after separating an access change, a price change and a change in the underlying content.
Compare the deployment boundaries before the features
AWS documents AI traffic monetization for CloudFront-associated WAF protection, using a machine-payment challenge and a facilitator-supported settlement flow. Its overview says the feature adds no charge beyond standard WAF pricing. That statement does not eliminate distribution, compute, payment-provider or operating costs elsewhere in the implementation. [1]
Cloudflare’s native Pay Per Crawl remains labelled closed beta in the documentation checked for this article. Cloudflare describes itself as Merchant of Record for that product. Native Pay Per Crawl and a separately built x402 Worker are different implementations; do not transfer eligibility, charging or recovery assumptions between them. [6]
| Decision | AWS WAF route | Cloudflare native route |
|---|---|---|
| Initial scope | CloudFront-associated web ACL and monetization configuration. | Eligible Pay Per Crawl account and zone. |
| Payment relationship | Document the selected wallet or connected-account route. | Document the native Merchant of Record arrangement. |
| Before rollout | Match the test environment to the settlement configuration. | Confirm beta access and the actual zone configuration. |
| Unproved by documentation | Buyer demand, identification accuracy, earnings and your deployment’s behavior. | |
A product can have a documented feature while your account, client or resource cannot use it. Keep that missing evidence visible. An architecture comparison is not a record of completed onboarding.
Trace the effective rule, not just the selected action
A publisher can select Charge and still block the intended request earlier. Cloudflare documents that WAF processing precedes native Pay Per Crawl. Its troubleshooting guidance also warns that upstream controls can affect requests without the reason being visible in the crawl-control view. [7]
Review the relevant request path with the policy owner. Which rule decides first? Is the resource already denied, redirected or transformed? Does a broad exception bypass a later restriction? Record the actual rule identifiers privately and retain non-sensitive aliases in a shareable worksheet. Do not resolve a revenue problem by blindly removing an access control.
Human access deserves its own case. AWS warns that its Monetize response is a machine-readable x402 challenge that standard human browsing cannot complete. Its pricing guidance therefore recommends limiting the action to the intended automated traffic. [4]
The commercial failure can be subtle: the paid route works for a development client, while an ordinary reader receives a challenge they cannot use. Test the reader route, the intended machine route and the deliberately disallowed route separately. Success for one does not establish the other two.
The DN record reports a declared rule conflict when the entered information says an earlier control contradicts the intended action. It does not discover that control or know which rule should prevail.
Keep payment, origin success and delivery separate
AWS’s documented lifecycle evaluates the rule, challenges the client, verifies authorization and fetches the resource. A successful 2xx origin response proceeds to settlement; origin 4xx/5xx responses are not charged. The documented flow serves the content after payment confirmation. These are stated implementation behaviors, not DN latency or reliability measurements. [3]
Cloudflare’s FAQ says charging events depend on successful responses and errors are not billed. It also treats recrawling as a new potentially chargeable access. That is different from assuming a client bought a permanent right to make unlimited requests. [10]
For a pilot, retain separate counts for challenged requests, payment attempts, reported settlement, successful response and the result the client could actually use. Choose stable identifiers and a cutoff. Retries belong to the same incident analysis even when billing treats them as separate operations.
A successful HTTP response is a useful technical boundary, but it cannot establish that the returned research was current, complete or worth its price. Define what a buyer is purchasing beyond transport success. For example, a dated data file should identify its observation period and revision policy, rather than presenting a successful download as evidence of freshness.
Publish the usage terms before the purchase depends on them
Payment instructions and usage rights answer different questions. AWS explicitly separates monetization from permission to use the content and describes RSL-based discovery through several surfaces. Its CloudFront example adds a licence link to origin responses, while warning that the WAF-generated 402 challenge will not contain that header. [5]
That distinction creates a practical review question: can the intended client find the applicable terms before it commits to paying? Record the terms version, resource scope and discovery route. If the only link arrives after the purchase, the buyer’s authorization process may lack information it was meant to consider.
Choose terms that describe the intended use and have the appropriate legal review. A machine-readable document is not proof that every client interprets it, accepts it or complies with it. Conversely, a paid request does not by itself establish permission for training, redistribution or publication of the entire response.
Keep policy changes versioned. When a price stays the same but reuse rights change, the commercial offer has still changed. A stored receipt should be linked to the relevant offer and terms, subject to the publisher’s actual recordkeeping and privacy obligations.
Use the pricing mechanism that actually exists
AWS documents a base request price and rule-level multipliers. The supported amount precision and minimum differ between self-managed wallets and connected accounts. Therefore a pricing policy must name its settlement route; a number copied from another configuration is insufficient. [4]
Cloudflare’s current price setup supports a default zone price plus optional dynamic content pricing. Its advanced guide explains an origin or Worker crawler-price response header and identifies where later response transformations are too late to choose that price. [9] [8]
There is a documentation nuance worth preserving: the older Cloudflare FAQ describes a single price for charged crawlers, while the more specific July 2026 setup and advanced pages describe dynamic content pricing. This article follows those implementation pages for content-price variation; it does not promise that every account has arbitrary per-customer contracts or negotiated pricing. [10]
The instrument intentionally asks no assumed traffic volume and computes no earnings. Before building a financial model, establish what constitutes a billable event, what the provider charges and which real buyers accept the offer. DN’s existing x402 economics guide addresses payment-cost denominators; this article addresses access-policy consistency.
Use an authorized pilot with explicit stop conditions
AWS describes CurrencyMode: TEST for self-managed CryptoConfig, using test networks. It does not support that mode for connected accounts, and it recommends test mode on non-production traffic. The same documentation describes classification as probabilistic. [2]
These details change the pilot plan. “Test first” is incomplete if the chosen account route cannot use that particular test mode. Resolve the documented environment before submitting requests or real payments. Keep the tested route, configuration and client version with the result; a test on another route is useful context, not proof of this one.
Define a small permitted case set: an intended paying client, an ordinary human request, a declined payment, an origin failure, a repeated request and a resource whose licence or price has changed. These are proposed cases, not tests DN ran. Add a documented way to stop the pilot and restore the agreed policy without widening permissions.
Choose stop conditions before observing revenue. Unexpected human challenges, charging outside the intended scope, unexplained payment states or missing evidence should prompt investigation. A handful of successful paid responses cannot establish that those exceptions never occur.
Use the DN Crawl Access Policy Record
The local instrument records one declared policy and identifies selected contradictions or unresolved evidence. It does not fetch your site, examine WAF settings, authenticate a crawler, read an account or change access. Use a nonpersonal label such as PREMIUM-ARCHIVE-A.
Its first synthetic example proposes charging through an AWS connected account in TEST mode, while an earlier rule blocks the request and human traffic may match. The record identifies those declared conflicts. It does not invent a success rate or decide which control to remove. The second example leaves account access, client support and test evidence unknown; the output stays incomplete.
A third, internally consistent example records a self-managed AWS test route, supported client and no selected conflict. The operational conclusion remains Not established. This is deliberate: choosing reassuring answers cannot turn an uninspected configuration into a verified deployment.
Allow and Block policies are also useful records. Payment-client and charging-event fields remain in the export but are not evaluated as payment conditions unless Charge is selected. A prior-rule conflict or unknown action remains visible regardless. The JSON is a portable statement of entered evidence and proposed decisions, not a server configuration or compliance certificate.
Choose the next business decision from observed evidence
A paid-access pilot should answer a commercial question with the smallest credible evidence set. Did eligible clients discover the resource? Which accepted the offer? Was the delivered resource useful? What support, reconciliation and content-maintenance work remained? Retain the observation window and the distinction between requests, buyers and purchases.
Separate demand changes from classification and access changes. If more requests appear after a new rule, the increase might reflect relabelled traffic or a retry loop. If fewer requests appear, that may reflect the intended restriction, an unsupported client or accidental denial. Neither direction alone identifies a profitable price.
The decision may be to continue a narrow paid product, keep a discovery route free, negotiate a different licence or stop the experiment. The useful result is a defensible next action tied to observed behavior. The publication of a 402 interface does not establish the end of advertising, an automatic market for every article or a recurring revenue stream.
Methodology and limits
The instrument receives a resource label and eleven categorical selections. It compares declared action, environment, earlier-rule behavior, human access, client compatibility, charging event, terms discovery and test evidence. Known contradictions take precedence over unresolved evidence; otherwise the result states only that no selected conflict was found. No weights, probabilities or financial forecasts are calculated.
Unknown is an explicit input, not a favorable default. Unsupported values and a blank resource label are rejected. Payment-specific conditions are evaluated only for a Charge policy. The exported inputs retain the other selections so later reviewers can understand the record. Every operational conclusion remains unestablished because the tool performs no inspection. Preserve the underlying source references and configuration version in your own controlled records.
The comparison covers the named documented implementations, not every x402 facilitator or custom Worker. Account eligibility, product availability, price rules and documentation can change. A self-reported test is not authenticated here; successful delivery does not prove lawful reuse or useful content. The record neither deploys controls nor recommends removing them. It supplies no demand, earnings, traffic-classification accuracy or security benchmark.
Machine-readable record schema · 1.0.0
This JSON Schema documents the downloadable record, including input provenance, missing-value handling and method versions. The creation timestamp records the calculation time; it does not certify when the underlying evidence was observed.
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "urn:dn:record:dn-crawl-access-policy-record:1.0.0",
"title": "DN Crawl Access Policy Record record",
"type": "object",
"additionalProperties": false,
"required": [
"schemaVersion",
"recordType",
"createdAt",
"articleSourceReviewDate",
"inputEvidenceStatus",
"inputBasis",
"exampleLabel",
"engineVersion",
"inputs",
"result"
],
"properties": {
"schemaVersion": {
"const": "1.0.0"
},
"recordType": {
"const": "dn-crawl-access-policy-record"
},
"createdAt": {
"type": "string",
"format": "date-time"
},
"articleSourceReviewDate": {
"const": "2026-11-13"
},
"inputEvidenceStatus": {
"const": "user_entered_not_independently_verified"
},
"inputBasis": {
"enum": [
"user_entered",
"illustrative_example",
"illustrative_example_modified"
]
},
"exampleLabel": {
"type": [
"string",
"null"
]
},
"engineVersion": {
"const": "1.1.0"
},
"inputs": {
"type": "object",
"additionalProperties": false,
"required": [
"resource",
"provider",
"action",
"mode",
"access",
"upstream",
"human",
"client",
"billing",
"license",
"test_evidence",
"purpose"
],
"properties": {
"resource": {
"type": "string",
"maxLength": 120,
"minLength": 1
},
"provider": {
"type": "string",
"enum": [
"unknown",
"aws_wallet",
"aws_connected",
"cloudflare_native",
"other"
]
},
"action": {
"type": "string",
"enum": [
"unknown",
"allow",
"charge",
"block"
]
},
"mode": {
"type": "string",
"enum": [
"unknown",
"proposed",
"test",
"live"
]
},
"access": {
"type": "string",
"enum": [
"unknown",
"confirmed",
"not_confirmed"
]
},
"upstream": {
"type": "string",
"enum": [
"unknown",
"conflict",
"reviewed_no_conflict"
]
},
"human": {
"type": "string",
"enum": [
"unknown",
"excluded",
"could_match"
]
},
"client": {
"type": "string",
"enum": [
"unknown",
"supported",
"unsupported"
]
},
"billing": {
"type": "string",
"enum": [
"unknown",
"success_only",
"failed_or_attempts"
]
},
"license": {
"type": "string",
"enum": [
"unknown",
"before_payment",
"after_payment",
"not_defined"
]
},
"test_evidence": {
"type": "string",
"enum": [
"unknown",
"not_tested",
"observed"
]
},
"purpose": {
"type": "string",
"enum": [
"unknown",
"search",
"retrieval",
"training",
"mixed"
]
}
}
},
"result": {
"type": "object",
"additionalProperties": false,
"required": [
"summary",
"metrics",
"findings"
],
"properties": {
"summary": {
"type": "string"
},
"metrics": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": false,
"required": [
"label",
"value"
],
"properties": {
"label": {
"type": "string"
},
"value": {
"type": [
"number",
"string",
"null",
"boolean"
]
},
"unit": {
"type": "string"
},
"note": {
"type": "string"
}
}
}
},
"findings": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": false,
"required": [
"level",
"text"
],
"properties": {
"level": {
"enum": [
"note",
"warning",
"error"
]
},
"text": {
"type": "string"
}
}
}
}
}
}
},
"allOf": [
{
"if": {
"properties": {
"inputBasis": {
"const": "user_entered"
}
}
},
"then": {
"properties": {
"exampleLabel": {
"type": "null"
}
}
},
"else": {
"properties": {
"exampleLabel": {
"type": "string",
"minLength": 1
}
}
}
}
]
}Related DN research
Frequently asked questions
Does returning HTTP 402 make an article earn money?
No. A compatible client must accept the offer and complete the applicable payment flow. A charge setting alone establishes neither demand nor net revenue.
Is Cloudflare Pay Per Crawl generally available?
The native product documentation checked on 13 November 2026 labels it closed beta. Confirm account access. A separately built x402 Worker is a different implementation.
Can I use AWS test mode with a connected account?
The checked AWS guidance supports CurrencyMode TEST only with self-managed CryptoConfig. It does not support TEST for connected accounts.
Why can a crawler marked Charge still be blocked?
An earlier control may deny the request before the charging layer. Inspect the actual rule path; the worksheet only records the conflict you report.
Does a paid crawl authorize model training?
Payment and permission to reuse content are separate questions. Identify the applicable terms and their legal scope; this tool gives no licensing opinion.
Does No selected conflict mean ready to launch?
No. It describes consistency among entered selections. The tool has not inspected the site, verified identity, observed a payment or approved a deployment.
Sources and review dates
- AWS: AI traffic monetization. Documented CloudFront/WAF availability and feature-charge boundary. Reviewed 2026-11-13.
- AWS: Getting started and test-mode constraints. Configuration, settlement-route distinction, probabilistic classification and restricted test mode. Reviewed 2026-11-13.
- AWS: Request, origin and settlement lifecycle. Documented successful-origin settlement and failed-origin boundary; no performance measurement. Reviewed 2026-11-13.
- AWS: Pricing configuration and human traffic. Route-specific price configuration, rule multipliers and machine-only challenge warning. Reviewed 2026-11-13.
- AWS: Communicating license terms. Terms discovery surfaces and absence of origin response-header policy from WAF402. Reviewed 2026-11-13.
- Cloudflare: What is Pay Per Crawl?. Closed beta, native Merchant of Record and conceptual payment access. Reviewed 2026-11-13. Source date: 2026-07-28.
- Cloudflare: AI Crawl Control with WAF. Rule order and conflicts; not proof of any deployed site configuration. Reviewed 2026-11-13. Source date: 2026-08-03.
- Cloudflare: Advanced configuration. URI exclusions and dynamic content pricing at the documented response stage. Reviewed 2026-11-13. Source date: 2026-07-28.
- Cloudflare: Set a pay per crawl price. Default price and optional dynamic content price setup. Reviewed 2026-11-13. Source date: 2026-07-28.
- Cloudflare: Pay Per Crawl FAQ. Error and recrawl charging; older single-price wording is read alongside the more specific current dynamic-pricing pages. Reviewed 2026-11-13. Source date: 2026-04-23.
Turn the finding into a decision
For a bounded review of the evidence and unresolved questions behind a specific operational decision, contact the DN Research Desk. Scope, feasibility, fee and delivery terms must be agreed before work begins.
Disclosure: This is independent DN editorial research. The instrument evaluates entered assumptions and does not independently verify your evidence or approve transactions. Illustrative examples are labelled. Any affiliate placement is identified where it appears; a commercial relationship does not determine the findings. No provider endorsement or guaranteed outcome is implied.
