Changelog

When a LinkedIn post was no longer public, LinkedIn sometimes showed its sign-up page in a language other than English. /v1/linkedin/search-posts could return that page as a post with author "LinkedIn". It is now recognised in every language and left out of the results.

• linkedin/search-posts: sign-up pages in any language are dropped from the results
• linkedin/post-details and post-transcript: a sign-up page answers 502 login_wall (retryable) instead of returning page text

When Reddit refused to show a search page on the first try, /v1/reddit/search answered 503 UPSTREAM_UNAVAILABLE. It now reads the page once more before giving up. A 503 is still possible, still costs 0 credits and is still safe to retry.

• reddit/search: one more read of the search page before answering 503
• Other Reddit endpoints that read Reddit pages get the same extra read
• 503 responses stay at 0 credits

Kwai post and user-posts returned a mediaUrlsExpireAt that was already in the past when the response was sent, which made working links look dead. Kwai's signed links don't carry an expiry we can read, so the field is now null; re-fetch rather than archive videoUrl. Instagram highlights and highlights-details now report mediaUrlsExpireAt, the earliest expiry of their signed image and media links, like the other Instagram endpoints.

• kwai/post, kwai/user-posts: mediaUrlsExpireAt is null (was a timestamp at or before fetchedAt)
• instagram/highlights, instagram/highlights-details: mediaUrlsExpireAt added (earliest signed link expiry); absent when the response has no media link

For a query with no TikTok Shop products, Shop Search could return unrelated products and charge 2 credits. Products that are not about the query are now dropped, so such a query returns 200 with products: [] and totalReturned 0 at 0 credits, as documented. serpHits counts only the links about the query. Searches with real matches are unchanged.

• tiktok-shop/shop-search: no matching products: 200, products [], 0 credits (was: unrelated products, 2 credits)
• tiktok-shop/shop-search: products not about the query are dropped from a results page
• tiktok-shop/shop-search: serpHits counts only links about the query

/v1/rumble/channel-videos answered 404 "Channel not found" for Rumble user accounts (https://rumble.com/user/name), because they were looked up under /c/name. User and channel URLs now both return the uploads, read from Rumble's public /videos pages (25 per page, newest first). A channel with no uploads is a 200 with an empty list at 0 credits.

• rumble/channel-videos: https://rumble.com/user/name URLs return uploads (was 404)
• rumble/channel-videos: empty channel is 0 credits (was 2)

On /v1/linkedin/company, every similarPages row came back with image: null because LinkedIn lazy-loads those logos on its public page. Each similar page now carries its logo URL. Price and the other fields are unchanged.

• linkedin/company: similarPages[].image is the page logo (was always null)

With fast=true, and when the RSS or fallback path served the page, /v1/youtube/playlist-videos left totalVideos out of the response. The key is now always there; it is null on those paths because they don't carry the playlist size.

• playlist-videos: totalVideos keyed on every path (null with fast=true)

A public English Reddit post sometimes came back with its body fully translated (title and comments stayed English), so a customer could not tell the text was not the author's. Reddit picks that translation from the viewer's country. Reddit reads now go through a US exit with English Accept-Language, so post-details and post-comments return the original body.

/v1/instagram/highlights returned only the first page of a profile's Story Highlights. It now follows the remaining pages, so albums past the first page are included too.

• instagram/highlights: highlights[] lists every album on the shelf, not just the first page

On channel-posts and channel-reels the author block of the account's own posts carried followers on some rows only, and user.postCount was usually null. Owner rows now take followers and postCount from the profile in the same response, and user.postCount is filled when Instagram's public page shows it.

• instagram/channel-posts: posts[].author.followers and author.postCount match user{} on the owner's own posts
• instagram/channel-reels: reels[].author.followers matches user{} on the owner's own reels
• instagram/channel-posts and channel-reels: user.postCount is filled when available

Since early October channel-details and profile-search often returned followers but left following and postCount null, on public and private accounts alike. Both counts are filled again on every profile Instagram shows them for.

• instagram/channel-details: following and postCount are filled again
• instagram/profile-search: following and postCount are filled again

On private accounts basic-profile returned followers but left following and postCount null, although Instagram shows both counts to logged-out visitors. Both are now filled, and private-account lookups return faster.

• instagram/basic-profile: following and postCount are filled on private accounts
• instagram/basic-profile: followingIsApproximate and postCountIsApproximate are set with them

facebook/search and facebook/company-ads return the ads from one Meta result batch (about 30). When a request asked for more than that, hasMore was false even though Meta reported thousands more matches. hasMore is now null in that case and the new truncatedReason field says library-page, so a spent batch is no longer reported as the end of the results.

• hasMore: true when nextCursor is present, null when Meta has ads past the batch, false only at the real end
• new truncatedReason: library-page when the batch is spent but Meta has more; null otherwise
• nextCursor and retrievableCount are unchanged

channel.thumbnail on YouTube search, channel-tab and hashtag video cards was always null even though YouTube shows the channel avatar next to each result. It is now filled. The ad-library docs also now name the advertiser fields that each library does not publish, so a null there is expected rather than a missed value.

• youtube/search: videos[].channel.thumbnail and lives[].channel.thumbnail are the channel avatar URL
• ad-library docs: advertiser.location is null in practice on facebook/search, linkedin/search-ads and linkedin/ad-details
• ad-library docs: advertiser.logo is null in practice on tiktok/search; google/company-ads publishes no logo, location or cta

When our first lookup path was slow, about 3 in 10 details responses came back with only the post id and the author's username: no likes, comments, caption or image. The backup path now reads the full post card Instagram publishes for embeds, so these responses include engagement, caption, carousel children and a full-size image. A username-only result is no longer returned as a success.

• engagement (likes or views, comments), caption and children are filled on the backup path
• thumbnailUrl is the full-size image on that path (was a 150px copy when present)
• publishedAt can still be null on the backup path

When a company changes its LinkedIn URL, the old address redirects to the new one (company/the-world-bank now redirects to company/the-world-bank-group). That redirect was often blocked, and the call ended in a 502 UPSTREAM_UNAVAILABLE after about 46 seconds. We now fetch the new address directly, and the old URL returns posts in a few seconds.

• Old company URLs that redirect to a renamed page return that page's posts
• No change to credits or the response shape

A failed hashtag lookup was remembered for every hashtag for 60 seconds, so a valid tag searched right after a misspelled one got an instant 502 login_wall. Failures are now remembered per hashtag, and a hashtag Instagram does not have returns 404 HASHTAG_NOT_FOUND instead of a login_wall 502.

• Negative cache is per hashtag: error.cacheScope is now input (was service)
• persistent=true needs 3 fresh misses on the same hashtag
• Unknown hashtags (no top and no Reels grid) are 404 HASHTAG_NOT_FOUND at 0 credits, decided before the slower fallback path runs

TikTok serves Brazil (region=BR) product pages only to in-market sessions. We were trying several page variants first and waiting ~50s before returning the same price-less row. BR lookups now skip those attempts and answer in about 20 seconds.

• BR products return title, images, stock, sold and seller name; price, description and rating are null with priceStatus=not_extracted
• Retrying a BR product in the same region will not return a price; the docs no longer suggest it
• Other regions are unchanged

music-posts was falling back to a secondary path for every sound: the native walk collected rows but was cut off at its own deadline and the rows were thrown away. The secondary path could not paginate yet reported hasMore=false, which read as "no more posts". The endpoint now serves TikTok's music feed only, returns short pages with a working nextCursor, and says plainly when it cannot tell whether more posts exist.

• Breaking: hasMore is now true / false / null. New pagination field: cursor (pass nextCursor), end (TikTok's feed for this sound is exhausted), unavailable (cannot tell — hasMore null)
• Short pages: TikTok serves this feed a few videos per page, so a call returns what lands within ~35s plus a nextCursor
• Breaking: no secondary path. When TikTok's feed refuses us the answer is 503 UPSTREAM_UNAVAILABLE with Retry-After (reason native_unavailable, page_miss or timeout) at 0 credits
• Breaking: limit is 1–50; anything outside that is a 400 instead of a request that timed out
• Error messages now state how long we actually waited
• Breaking: timings.apifyMs and stages.apifyStatus were removed

When a company URL pointed at a slug LinkedIn itself answers with HTTP 404 (pages move, or a duplicate registration was removed), the endpoint reported a retryable 502 "unavailable. Retry shortly." — but no retry can fix a page that does not exist. LinkedIn's own not-found answer is now surfaced as a definitive 404.

• Dead company slugs now return 404 "Company not found on LinkedIn" with a hint to check the URL, instead of a retryable 502
• The 404 verdict comes only from LinkedIn's own answer for the page — network trouble on our side keeps the retryable wording
• Failed lookups remain unbilled (0 credits), as before

Passing a facebook.com/share/… link to /v1/facebook/comments or /v1/facebook/comment-replies skipped the share-resolution step that /v1/facebook/details already had, so the extractor read the share interstitial instead of the post — and could answer 200 with zero comments or replies, billed, even when the target was unreadable. Share links are now resolved to the canonical post URL first, and a share we cannot resolve returns the same clear 400 as details, unbilled.

• share/… URLs on both endpoints resolve through the same ladder as facebook/details and the response echoes the canonical post URL
• Unresolvable share links now return 400 UNRESOLVED_SHARE_LINK at 0 credits instead of a billed empty 200
• comment-replies answers "0 replies" only when the parent comment is visible and Facebook itself reports zero replies; an unreadable page falls through to the extended fetcher instead of asserting an empty thread

facebook.com/share/p/… links whose target is a post in a public Facebook group were answered with UNRESOLVED_SHARE_LINK even though the share resolved cleanly — the resolver found the canonical group-permalink URL but our URL classifier did not recognise the /groups/<group>/posts/<id> form and discarded it. Group permalinks are now first-class post URLs.

• share/p/ links to public group posts resolve and serve full details (text, engagement, media) — the two links failing this morning now both return data
• Direct /groups/<group>/posts/<id> and /groups/<group>/permalink/<id> URLs are accepted by facebook/details without needing a share link at all
• share/r and share/v links were unaffected and keep working as before; unresolvable shares still return the same clear 400

Top Ads captures run through country-matched fetch pools, and an unhealthy pool for one country could fail the whole request even though the leaderboard itself was reachable — a burst of 503s this morning came from exactly that. A failed capture now retries once through a known-good pool before giving up (the country you query is controlled by the request itself, so results are unchanged).

• One unhealthy regional pool no longer turns into a customer 503 — verified live: a UK request that failed on the first capture was served on the rescue attempt
• Both attempts share one overall time budget, so worst-case latency stays at today's ceiling (~95s) and typical rescues land in 35-50s
• Failures remain free and retryable; the last-good fallback page (up to 6h) still serves when both attempts miss

Some LinkedIn company pages show no posts to logged-out visitors — often a stub or duplicate registration while the organization actually posts from another page. Those requests used to burn up to 110 seconds across serial fallbacks and then claim 'Retry shortly' for a state a retry cannot fix. The rescue search now runs in parallel with the extended fetcher and respects its time budget, and confirmed-empty pages get an accurate error.

• When the company page carries no public posts, the rescue search starts immediately in parallel instead of last in line — successful rescues and failures both return tens of seconds sooner
• The rescue search's engine sweep now honours one overall time budget (it could previously run double and push requests into the 110s wall), and it no longer spends time fetching other authors' posts that merely mention the company
• New honest error when the page is confirmed to show no public posts: 'LinkedIn is not showing this company's posts to logged-out visitors right now… verify the company URL' — no more 'Retry shortly' when retrying cannot help
• Transient failures keep the retryable wording, and all failures still cost 0 credits

Instagram serves some existing profiles as a logged-out “Restricted profile” wall (age- or region-gated) that withholds every data field. All lookup sources came back empty and the endpoint answered “Instagram profile lookup failed. Retry shortly.” — but no retry can pass a login wall.

• The rendered profile page is now checked for Instagram's restricted-profile wall; when it is the reason nothing parsed, the response is 403 profile_restricted with retryable: false and a message explaining the platform limitation
• Real data still wins: the verdict is only returned when every source came back empty — an account that any fetcher can read keeps its normal 200
• Genuinely transient lookup failures keep the retryable 502, confirmed-gone accounts keep the 404, and error responses stay unbilled

Spark/Non-Spark is not on TikTok's public Top Ads data — only the extended fetcher carries it, and that fetcher is currently disabled. Every adFormat request was deterministically unservable, yet it returned a retryable “temporarily unavailable” 503 in ~460ms; one caller retried the same request in bursts for 20 minutes with zero chance of success.

• adFormat requests now return 503 filter_unavailable with retryable: false and a message that says exactly what to do: omit adFormat to get the native leaderboard now
• Still 0 credits, and the gate fires before any fetch or cache machinery — no stale page can dress an unservable filter as served data
• When the extended fetcher is re-enabled, adFormat serves again with no client change; parameter docs updated to state the contract

When the native fetch was rejected at the door and the extended fetcher was parked, the endpoint gave up in ~300ms with a retryable 503 — even when a minutes-old page for the very same parameters was sitting in cache (26 Sep: three 503s while a 27-minute-old page went unused).

• A 5xx miss is now rescued by the same parameters' last-good page from the past 6 hours, labelled stale: true + cached: true with the original fetchedAt, at 0 credits — the leaderboard is a 7/30/180-day aggregate, so a recent page is still the leaderboard
• With no last-good page the honest retryable 503 at 0 credits is unchanged, and validation errors are never masked by stale data
• Same contract as facebook profile-posts / group-posts last-good rescue

A post URL whose page slug does not match the post's owner (or a deleted/restricted post) renders Facebook's 'content isn't available' page — which was classified as a transient miss and answered 503 'retry shortly' forever. Retrying such a URL can never succeed.

• When both native renders serve Facebook's dead-content page and the extended fetcher cannot read the post either, the answer is a non-retryable 404 explaining that the post was deleted, restricted, or the URL's page slug does not match the post author
• summarize now has the same rescue ladder as details: a transient native miss gets one extended-fetcher read before any error verdict
• A login wall never counts as the dead-content verdict, and a single ambiguous render keeps the honest retryable 503

The public timeline embed misses for two very different reasons: a transient outage (retry helps) and an account that has simply never tweeted (retry can never help). Both collapsed into the same 502 'Retry shortly' — an account created in 2016 with zero tweets sent callers into a retry loop that could never succeed.

• On an embed miss the account's public profile is checked (~1s); an explicit zero tweet count returns 200 with tweets: [], truncatedReason no-tweets and an explanatory note — at 0 credits
• Accounts that do have tweets keep the honest retryable 502 when the embed is withheld or the fetch flakes

An existing-but-private Threads account has zero visible posts on its hydrated page, which collapsed into the generic 404 suggesting the handle might be wrong. The page itself carries the account's private flag — that verdict is now read and reported.

• A private account returns 200 with posts: [], isPrivate: true, the author card (username, display name, avatar) and an explanatory note — at 0 credits
• The Instagram bio-link handle remap is skipped for private accounts: the handle is right, so a lookalike swap would be wrong
• Dead handles and public zero-post profiles keep their existing 404 verdict

Group feeds render through a single headless fetch whose yield is inherently variable — the same group polled repeatedly flip-flopped between 200s and 502/503s (26 Sep: three groups, a dozen calls, roughly every third one failing). Failures were honest but useless to a caller who got a full page minutes earlier.

• Successful pages are remembered for 6 hours per group+sort; when a fresh fetch fails or times out, the last good page is served as a stale 200 (stale:true, source cache) at 0 credits — the same contract profile-posts and marketplace-search already have
• A first-ever fetch that fails still returns the honest retryable 5xx — stale pages are only ever real pages we served before

TikTok removed the public Creative Center song chart in its TikTok One migration (re-verified 26 Sep: the music page lands on a hub with no songs tab and the chart API answers 40101 anonymously), so the endpoint's honest 503 stands. But the hourly recovery probe formed that verdict from the anonymous session-check call — which answers the same deny code even when chart APIs are public — so a restored chart could never have been detected.

• The probe now inspects the full page traffic: chart rows win immediately (recovery auto-resumes 200s), a deny answered by the chart API itself is a real login gate, a loaded page with no chart call at all is the removed surface, and nothing loading is a transport miss that still tries the extended fallback
• Customer-facing behavior is unchanged during the outage: 503 CREATIVE_CENTER_UNAVAILABLE, retryable:false, 0 credits, one ~20s rediscovery probe per hour with instant replays in between

The earlier newest-first fix covered the extended fetcher path only. The native search leg trusted the search engine's date ordering — but that hint only reaches one of the two engines and concurrent post hydration shuffles the order anyway, so a sort=date page could come back as Sep, Feb, Mar, Jun.

• Both fetch paths now sort the batch by real publish time before slicing: the post's activity-id snowflake first, with the row's publishedAt timestamp as fallback
• Batch cache key bumped so pre-fix unsorted batches cannot serve inconsistent page 2s

The no_shop verdict for ttSeller:false accounts trusted a single empty shelf fetch. Affiliate creators are ttSeller:false by design yet still list showcase products — a transient empty run could brand them no_shop and bill a credit for a wrong answer.

• The profile's own commerce bit now gates the verdict: when TikTok flags the account as a commerce user and the shelf comes back empty, the response is an honest 502 SHOWCASE_UNAVAILABLE with 0 credits instead of a guessed no_shop
• Genuinely non-commerce accounts (both flags false) keep the 200 no_shop verdict at the 1-credit looked-and-empty rate — verified live against a non-seller profile from three independent signals

On the extended path, publishedAt was passed through from the upstream fetcher's own locale-rendered date string — a 25 Sep page showed posts published at 21:45 on a response fetched at 20:06 UTC (+2h skew, no timezone marker). The same pages also arrived as shuffled ascending runs instead of the recency order sort=date promises.

• publishedAt now derives from the row's epoch timestamp rendered in UTC (the post's activity-id snowflake confirms it); the vendor's local string is only a fallback when no timestamp exists
• sort=date pages are now sorted newest-first before slicing — the cursor re-slices the same sorted batch, so page 2 stays consistent
• sort=relevance ordering is untouched — that ranking belongs to the search engine
• timings now include serpMs and hydrateMs on the extended path too: 70 of 74 seconds used to be invisible when the native leg burned its budget before the fallback

Salt Lake City was missing from the Marketplace city directory, so its searches were snapped to the nearest known metro — Las Vegas, 360+ miles away — and the response still echoed the requested location. Results looked scoped to Salt Lake City but were another city's listings (or an empty page when the Vegas search found nothing).

• Salt Lake City now resolves to its own verified Marketplace hub — searches return Utah-metro listings (verified live: Midvale, Ogden, Provo, Draper results)
• Bare city names like 'salt lake city' or 'chicago' now hit the directory directly when the name is unambiguous — no more geocode detour (resolve dropped from ~700ms to ~1ms)
• A real US city with no scopeable hub within 90 miles now returns an honest 422 UNSUPPORTED_LOCATION naming the nearest supported metro, instead of silently answering with another city's listings
• Genuinely empty searches are unchanged: a Facebook-confirmed zero-hit still returns an honest empty page

Three sort=date searches on 25 Sep fell through to the extended fetcher and failed after 60-80 seconds: the public sort value was passed to it raw, but the fetcher's vocabulary is relevance/date_posted, so every fallback died with an invalid-input error. That raw error — naming the vendor and its payload — then surfaced verbatim as a 500.

• The public sort=date value is now mapped to the extended fetcher's date_posted before the call — sort=date searches succeed on both paths (verified live)
• An upstream fetcher failure now returns the honest 502 UPSTREAM_UNAVAILABLE envelope with a generic retry message — the raw vendor payload stays in server logs only
• The request log now records these as 502 UPSTREAM_UNAVAILABLE too; it used to say 500 while the customer actually received a 502, so the dashboard disagreed with the wire
• Failed calls were already unbilled (0 credits) — that behaviour is unchanged

In one degraded window (25 Sep 11:21-11:36 UTC) 14 of 32 requests 502'd at ~20s: TikTok's mobile API was answering in 30+ seconds (over its 16s leg budget) while the Apify actor's slow tail overran its own 16s cutoff — and the route gave up with ~100 seconds of its 125s wall unused. The same handles served 200s minutes later once the actor finished; the 16s cutoff never kills the run server-side.

• On a full miss the route now rejoins the still-running actor run (up to 45s) or reads one that finished right after the cutoff, instead of paying for a duplicate
• The rescue runs only on the miss path — successful races are exactly as fast as before
• timings gains rescueMs and apifyStatus: 'rescued' so a saved request is visible; a genuine full outage still returns the honest 502

share/r/ links resolve through Facebook's own canonical, which is the videos-slug form (/<page>/videos/<slug>/<id>/). That form renders a watch-style page that the native ladder reads as 'no node' and the Apify rescue as a bare Video node with an id-only owner — so three live requests on one share link served pages with no author name, no caption and no shares, while the /reel/<id>/ form of the same video carried them all.

• A share/r/ canonical in the videos-slug form is rewritten to /reel/<id>/ (the reel id is the slug URL's trailing segment) before fetching
• The rewrite also applies when reading the 30-day resolve cache, so entries poisoned before this fix heal on their next hit — no cache surgery
• Non-reel share links keep whatever canonical they resolved to

When the native render misses and the Apify rescue serves a reel, two fields were being lost in shaping. The scraper stamps the numeric page id as pageName on vanity-less pages — and that id outranked the real name sitting in video_owner.name, so displayName came back as '61584242948204' instead of the page's actual name. And shares arrive only as the compact token ('41K'), which the integer parser silently dropped to null.

• A digits-only pageName no longer masks the real author name (verified against the live actor payload: displayName now 'Whisprs "', not the page id)
• Compact share counts parse like the native ladder's (41K → 41000) with an honest sharesIsApproximate: true; exact integers stay exact and unflagged
• Views stay null on the actor path — the payload genuinely carries no view count, and an honest null beats a guess

25 Sep 09:18 UTC: one more 88.4s 503. The concurrent plain safety net shipped on 24 Sep fired a single shot at t0 — when that one shot flaked (plain misses are per-session random, like the render ones) a stalled grown render left ~70 seconds with no rescue in flight. Its twin case at 24 Sep 16:47 UTC, where the shot landed, came back 200 in 88.5s.

• The safety net now keeps a fresh plain attempt in the air until the request window is spent (capped, so an instant-error upstream cannot run up Decodo spend)
• Small-limit (clicks=0) requests get the same net beside their headless retry pair — previously they had no rescue at all after the primary plain shot missed
• Happy path unchanged: a landed grown render still cancels the net immediately; verified live at 33-49s with zero extra attempts

country=India had never returned a page (six straight 502 fetch_empty at ~78s over 23-24 Sep). Two root causes: the country-level IN proxy pool cannot finish a headless render of instagram.com/reels at all (systematic read timeout at the full wall, 9/9 across prod and probes), and the old serial listing walk let two of those stalls eat the whole fetch window so the Explore page never even got a try. Indian CITY egress pools render the same India-localized page in 15-17 seconds.

• Listing renders now go out as parallel /reels + /explore/reels pairs (two rounds) — one stalled arm can no longer starve the other
• India rides a city-pool geo ladder: Mumbai egress first, the country pool as the second-round retry; hydration reuses whichever egress actually served the listing
• Verified live: India 7 reels in 23s (was a guaranteed ~78s 502); the same pair design also covers the Turkey-style empty-first-render misses

Instagram's login/WAF gate on explore/tags walled off every native tier (four 502 login_wall rows on 24 Sep), and a wall also gates the Decodo rung by design — so the whole ladder died with the session pool. The managed Instagram rung now answers the same hashtag sections feed (top for posts, clips for Reels) when the native listing misses outright. Both hashtag-search and reels-search share the rescue.

• Native listing miss → managed hashtag grid: raw media sections mapped by the exact same code path as a session page
• Media rows arrive complete (author, caption, counts), so the page ships without hopping dead sessions for enrichment
• Verified live with the four failing tags — #trendalert, #reeltrend, #newtrend, #trendingreels: 20 posts each in 14-22s (was 502 login_wall)

LinkedIn lazy-loads SERP creatives: the real media.licdn URL sits in a data-delayed-url attribute while src holds a placeholder ghost. The extractor read only src, so most cards shipped with an empty media array even though the images were on the page (one audited SERP carried 48 delayed URLs against 16 hydrated src). Separately, any card with media was labelled 'Single Image Ad' — including video posters and document pages, whose CDN paths name the creative kind.

• SERP extraction reads data-delayed-url before src — live media fill went from near-zero to 20/20 (microsoft/US) and 22/24 (Forex/DE)
• adFormat derives from the creative URL shape: videocover → Video Ad, ads-document-cover → Document Ad, multiple images → Carousel Ad; the same inference backs /linkedin/ad-details when the page carries no About-the-ad label (LinkedIn's own label still wins)
• advertiser/cta/paidForBy stay null on search rows by design — the SERP card does not carry them; hydrate via /linkedin/ad-details as the platformNote says

After the plain-first ladder shipped, five more 503s appeared with one shape: all on limits > 25, all at exactly ~88s. The grown browser render had stalled to its full 84s wall, leaving 11s — too little for the retry pair (needs 30s) and too little for the serial plain rescue (a plain fetch takes 3-12s, it got 8). The plain fetch now starts at t0 beside the browser render, so its first ~25 cards are in hand by ~15s no matter what the render does.

• Grown requests (limit > 25) start a cheap no-JS fetch in parallel with the browser render; it is discarded when the render lands
• When every browser render misses, the pre-fetched plain page ships the first ~25 cards with an honest hasMore=true instead of a 503
• Verified live against the exact failing shape: Forex/DE limit=48 with all three browser renders down — 24 ads in 83.5s (was a guaranteed 88s 503)

When our Instagram session cookies die (roughly weekly), the timeline feed 401s on every proxy tier, the tagged feed hits a login wall and the highlights tray rate-limits — a public profile answered 502 on the next feed page, a 90s timeout 503 on tagged posts, and a definitive-sounding 422 on highlights that actually had six live albums. All three surfaces now fall back to the same managed Instagram pool that already rescues channel-reels and comments, returning identical raw payloads through the existing mappers.

• Timeline feed pages (channel-posts pagination), the tagged feed and the highlights tray each gained the managed rung; public cursors pass through unchanged
• The tagged-posts native tier walk is time-boxed (dead sticky sessions burned ~2m45s of redirect loops — past the route deadline before any fallback could run)
• Tagged-posts profile resolve skips the session pool (same reasoning as highlights: dead-pool redirect loops burned ~72s of the 90s deadline)
• Author-stats enrichment is budgeted — when the pool is dead you get lean posts instead of a 503
• Verified live on the reported profile: next feed page 12 posts, 15 tagged posts, all 6 highlight albums with titles

A photo permalink 400'd as an invalid URL: the classifier knew permalink.php and story.php but not photo.php?fbid=, /photo/?fbid= or /page/photos/set/id — and the details parser only looked for Story nodes while photo pages embed a Photo node instead. Both layers now understand photos: the same single render serves caption, author, reactions, comments and the full-size image, and the feedbackId works with /v1/facebook/comments.

• All three photo permalink forms classify as posts and pass the details gate
• The Photo node's container story supplies the caption, author and canonical post URL — no second render needed
• A photo id that doesn't match any node on the page stays an honest retryable miss, never another photo's data
• Bonus: share links that land on photo URLs now resolve instead of failing as unresolved

During a degraded proxy-pool window (24 Sep 10:55–11:01 UTC) LinkedIn search answered three ~80s 503s interleaved with 200s of the very same query — the misses are per-session random, not systemic, yet the ladder retried with a single session and one more unlucky draw ended the request. The retry round now renders through two parallel sessions and serves whichever paints the library, cutting the joint-miss rate quadratically. The extra render is only spent when the first attempt already missed.

• Verified live: a first-attempt miss on a heavy advertiser recovered via the parallel pair — 20 ads in 87.5s where the single-session retry had answered nothing
• When both arms paint, the one with more ads wins; failures remain 0-credit and retryable
• Residual: a first attempt that stalls to its full 80s wall still can't fit a retry inside the 95s deadline — that rare case stays an honest retryable 503

When the first-page render missed, the pagination rescue could capture TikTok's own definitive answer — code 0, zero ads — and still throw it away, because zero rescued rows never beat an empty pool. Five zero-hit queries burned ~82 seconds each and answered 503 upstream_unavailable in one afternoon, while identical queries minutes later answered an honest empty 200 in 15 seconds. A code-0 empty answer from either rescue arm is now always a 200. The rescue round also runs both capture modes in parallel (they fail independently), so a 503 now requires three independent capture misses instead of two sequential ones.

• Verified live on all five failing queries: four zero-hits answered empty 200 in 13–19s, the fifth served 24 rows — none returned 503
• Rescue page-1 rows ship with honest hasMore instead of being dropped when pagination misses
• The extra parallel render costs only on the miss path; the direct JSON API stays fenced off (TikTok's per-request x-ccl-str signature answers 421 to every replay — verified across datacenter, residential, and direct egress)

Some videos sign audio-only stream URLs that answer 403 on every egress, while the combined audio+video stream from the very same player response downloads fine. The extractor treated the 403 as a dead end, burned its whole 20-second budget walking fallback clients that couldn't help, and answered 502 audio_extract_failed — the yt-dlp rescue never got a turn. The native rung now falls back to the muxed stream (video is stripped during the speech re-encode), the extract budget grew to 45s sized against measured ASR times under the 110s route deadline, and the fallback rungs must leave the yt-dlp rescue a reserved slice instead of starving it.

• Verified live on the failing video: audio-only 403 → muxed stream 200 in 2s, full 77s of audio transcribed end-to-end in 22s
• Interrupted yt-dlp downloads (.part files) can no longer be served as if they were the full audio
• Failures remain 0-credit; the 502 now only fires when every rung — including yt-dlp — actually ran and missed

After the tab-first ladder reorder, both light hops (first-paint home, then the /posts tab) ran under the same 22s deadline. Production showed renders of heavy pages like nasa spiking past 22s on both hops in the same request (listing phase 44s), landing a 503 that the failure marker then replayed fast. The home hop now gets 20s, the /posts tab hop gets its own 28s window sized to what's left of the budget, and old failure markers written before the wider window are versioned away.

• Home first-paint 20s + /posts tab 28s fits the 55s listing budget with margin for the details fetch
• Failure-marker cache key bumped so pre-fix fast-503s can't outlive the fix
• Deadline arithmetic is unit-tested against the observed 12–19s (calm) and 22s+ (spiky) render times

TikTok's One Creative Suite migration removed the public Creative Center song chart: the music page now redirects to a trends hub with only hashtag/video/creator tabs, the chart API answers 40101 no-permission to anonymous callers, and the extended actor scraping the same surface returns nothing (all verified 24 Sep). The endpoint already refused to substitute For You sounds; it now names the removal explicitly and remembers the outage for an hour instead of five minutes, so at most one 20–30s rediscovery probe runs per hour while auto-recovery stays in place should TikTok restore the chart.

• 503 CREATIVE_CENTER_UNAVAILABLE message now explains the chart was removed in the TikTok One migration (retryable: false)
• Persistent negative cache extended 5min → 1h — prod had burned eight 20–34s scrapes across 22–24 Sep rediscovering the same removal
• Last-good chart pages still serve as stale 200s when cached; all failures remain 0-credit

A stalled proxy render can run ~25s past its granted budget (read drain + wall cap). The retry attempt granted up to 55s whenever 30s remained on the clock, so a shell-page miss followed by a stalled retry walked one request to 113s against the 95s deadline before answering 503. Every attempt's grant now subtracts the stall overshoot from the time remaining, so the ladder mathematically cannot exceed its deadline.

• Root cause from prod 24 Sep: a 251-byte proxy shell at ~33s, then a full 55s retry grant whose stall burned 80s
• Attempts too small to fit a render plus its overshoot are skipped instead of started
• Same two-attempt retry contract — one blip still never 503s a search

Asking for more than 12 TikTok ads ran two 35-second first-page render attempts; when the proxy pool was slow, each stall overran its grant and together they ate the whole budget, then the route declared the library down (503) even though the pagination capture path was healthy in the same minute. The ladder now spends at most one tight ~15s probe on the first page and, when that misses, hands the remaining budget to the pagination capture as a rescue instead of giving up.

• Verified live: a first-page miss now recovers via pagination — 24 rows in 44s where the old ladder 503'd after 72s
• Zero-hit answers from TikTok's own API are still an honest empty 200, never a rescue
• Failures remain 0-credit and retryable

The episode archive interleaves RestrictedContent stubs (geo-blocked episodes) among real rows — Joe Rogan's page 1 opens with one. The stubs are unusable and get dropped, which shorted the page (limit=3 → 2 rows), and the offset cursor advanced by rows returned rather than archive entries consumed, so the next page re-fetched — and could re-serve — rows around the stub. Pages now backfill from the next archive slice to fill the requested limit, and cursors advance by entries actually consumed.

• Dropped RestrictedContent stubs are backfilled — limit=3 returns 3 episodes again
• nextCursor advances by archive entries consumed: no duplicates, no skipped episodes across pages
• Verified live on JRE: pages 1–2 are consecutive (#2557…#2552) with zero overlap
• Response-cache version bumped so short cached pages cannot serve

Both endpoints rode a single scrape-pool attempt with a tight timeout. The pool answers reddit JSON in 2–5s normally, but an individual request occasionally hangs well past any sane cap — that lone coin-flip turned into 503s on /search (the 6s attempt died) and 502s on /post-comments (the 15s comments.json fetch ReadTimed-out on a live thread). Both paths now retry once with a fresh request, which reliably lands.

• search: two scrape attempts budgeted inside a raised deadline (12s → 25s); direct-fetch rungs unchanged
• post-comments: comments.json fetch retries once (15s + 12s) after the authoritative existence probe
• Verified locally 3/3 on both paths against the previously failing thread and query

A single degraded page render that resolved no reel ids, combined with an actor miss, was treated as proof that the profile has no public reels — a 200 with noPublicReels: true and zero rows. Absence-of-evidence is not evidence: the affirmative 200 now requires either the actor's explicit empty answer or a two-render native confirmation. Anything less stays a retryable 5xx at 0 credits.

• noPublicReels: true requires affirmative evidence (actor-confirmed empty or double native confirm)
• Degraded single-render zero-id results no longer masquerade as an empty profile

When the native listing missed, a web-profile-info timeline fallback served whatever videos sat in the profile's ~12-post timeline edge — sometimes just 2 — and canceled the richer scrape/actor race that was still running. A limit=25 ask got 2 rows and a posts-style cursor. The thin timeline result is now held as a last-resort partial instead of served eagerly: the full ladder keeps running, and the thin set ships (flagged partial) only if every richer rung misses.

• Timeline fallback serves immediately only when it can fill min(limit, 12) rows
• Thinner sets are kept as a partial-of-last-resort while the scrape/actor race finishes
• Served thin sets are flagged partial: true with a wpi_thin stage marker
• Response-cache version bumped so cached 2-row pages cannot serve

The listing ladder tried a scrolled home-page render as its second hop. On heavy pages that render mostly dies in a ~40s ReadTimeout and eats the whole 55s budget, so the third hop — a light /posts tab render that actually paints post permalinks in ~19s — never ran. nasa returned 503 after ~53s on every attempt, and the failure marker then served fast 503s for the cache window. The /posts tab hop now runs second, and the light-hop deadline was raised from 16s to 22s to cover measured 12–19s first paints.

• Hop order is now: first-paint home → light /posts tab → scrolled home (last, only if time remains)
• Light-hop deadline 16s → 22s — real first paints of heavy pages were being cut off mid-render
• Verified live: nasa now answers 200 with hydrated post data in ~20s instead of 503 at 53s
• Response-cache and failure-marker versions bumped so pre-fix 503 markers cannot fast-serve

Comments beyond Instagram's 2-row logged-out preview depended on our own session-cookie pool, whose cookies die roughly weekly — when the pool was down, every limit>2 request answered 503 SESSION_UNAVAILABLE. A managed logged-in rung (HikerAPI /v2/media/comments) now pages the same raw comment shape with no cookies of ours: ~15 rows per hop with a real resume cursor. The session pool still runs first when alive; the 503 remains only for the rare case where every rung misses.

• Full comment threads page again during session-pool outages — verified live: 15 rows + working second page on an 820-comment post
• New cursor form {mediaPk}:h:{pageId} so managed pagination can never be mistaken for a session resume token
• Existing session cursors are honored by the pool only — the managed rung won't restart page 1 under a foreign cursor (no duplicates)
• timings.hikerMs exposes the new rung; failures stay 0-credit and retryable

Facebook's modern profile URLs come in a name+id pseudo-slug form (facebook.com/Amber-Daniels-6159…) alongside the plain numeric form (facebook.com/6159…). The listing scraper treated the pseudo-slug as a vanity slug: it fetched /{pseudo-slug}/reels — a degraded page — and skipped the numeric /videos permalink mining entirely. The same profile that listed 21 reel ids via its numeric URL returned a single reel via its pseudo-slug URL. Both URL forms (and /people/Name/id/ URLs, whose display-name segment had the same problem) now resolve to the numeric profile id before listing.

• Name+id pseudo-slugs (…-61590755182701) and /people/Name/id/ URLs resolve to the numeric id for /reels and /videos listings
• Numeric permalink mining now runs for these profiles, so the full recent grid is found
• Applies to both /v1/facebook/profile-reels and /v1/facebook/profile-posts; response-cache versions bumped so stale 1-item rows cannot serve
• Vanity slugs (nasa, blink-182) are untouched — only a trailing 10+ digit id run triggers the rewrite

Two deeper causes behind the remaining upstream_unavailable bursts. First, the HTTP client's read timeout never bounded total call duration: when the scrape provider kept the connection warm while its headless browser ground on, a 40s-grant pagination attempt ran 87 seconds and ate the entire request budget — every rung behind it starved. Every scrape call now carries a hard wall-clock cap, so a time grant means what it says. Second, the ladder opened with the flaky pagination rung even though most polled queries answer empty or with a single page. The ladder is now page-1-first: the reliable capture rung answers zero-hits in ~12s, pagination only runs when page 1 proves there are more ads, and a pagination miss ships the page-1 pool as an honest partial (truncated: true, hasMore: true) instead of a 503.

• Hard asyncio wall cap on every Decodo scrape call — a trickling connection can no longer run 2x past its grant
• limit>12 searches fetch page 1 first: zero-hit queries answer in ~12s instead of burning 40–90s of pagination
• Pagination failure degrades to the page-1 pool with truncated/hasMore flags — no more 503 when TikTok answered
• Fast-path (limit≤12) grants bounded to 35s so a degraded window cannot eat the retry's budget

With limit above 12 the search ladder ran two flaky Decodo paginate attempts back-to-back (~40s + ~26s of a 87s budget) before falling back to the fetch_resource rung that powers the fast path — leaving it an ~18s window that rarely landed. During degraded Decodo windows (21–23 Sep) identical queries flipped between 200s at 56–70s and 503 upstream_unavailable at ~90s, purely on the coin-flip of the second paginate attempt. The second paginate attempt now yields whenever it cannot leave the fallback a real (≥35s) window, so the rung that actually answers gets the budget instead.

• Second xhr-paginate attempt skipped when it would starve the fetch_resource fallback below ~35s
• Fallback still returns an honest partial pool (has_more/truncated flags) when pagination fails
• 503 upstream_unavailable remains retryable with 0 credits charged — but should now be rare instead of hourly

profileImage on profile-region is a signed tiktokcdn URL whose x-expires stamp lands ~2–3 days after fetch. Every other TikTok profile/video surface ships mediaUrlsExpireAt next to signed links; this endpoint shipped the dying URL with no warning, so a client caching the avatar past the stamp got a dead image with no way to know when it broke. The route now parses the x-expires stamp into mediaUrlsExpireAt, and the response-cache version is bumped so pre-fix rows without the key cannot serve.

• mediaUrlsExpireAt parsed from profileImage's x-expires (null when the URL carries no stamp)
• Cache version v9 → v10 — cached rows from before the fix are recomputed, not served without the key
• regionSource/regionConfidence semantics unchanged: tiktok = authoritative (confidence null), inferred = estimate with its grade

LinkedIn's experience location line mixes geography with the workplace arrangement — "Gurugram, Haryana, India · On-site", or just "On-site" when the member set no city. The profile endpoint passed that line through as location, so a live profile shipped an experience row whose "location" was On-site. Arrangement labels are now split out: experience rows carry location (geography only, absent when the member set none) and workplaceType (On-site / Remote / Hybrid, only when LinkedIn shows one).

• experience[].workplaceType: "On-site" | "Remote" | "Hybrid" — split from the location line, absent when LinkedIn shows none
• experience[].location keeps geography only; a bare arrangement label no longer masquerades as a place
• Combined lines ("City, State, Country · Hybrid") map to both fields

TikTok trims the seller's stray whitespace on each SKU's property pair but ships saleProperties with it intact — a live US phone-case PDP returned "iPhone 11 Pro Max " / "iPhone X " / "iPhone XR " / "iPhone XS " in the variant axis while every SKU said the trimmed name, so a client joining SKUs to the axis by value name silently lost 4 of 32 models. Both sides now trim names and values; whitespace-only axis values are dropped.

• skus[].saleProps propName/propValue and saleProperties[].name/values[].name are whitespace-trimmed to the same form
• Verified live on the reporting PDP: 32 axis values ↔ 32 distinct SKU values, zero unjoinable names
• IDs, prices, stock and everything else pass through unchanged

The 13:36–13:49 UTC wall burst walked 15 shortcodes through a fully walled ladder (GraphQL, embed and Decodo all gated) at 41–54s each. The Apify actor's not_found verdict shipped one false 404 for a live, public, verified account's carousel (hbro__), and four genuinely deleted posts spent the full deadline to die as retryable 502 fetch_exhausted — inviting pointless retries at dead URLs. HikerAPI's logged-in media/by/code answered all 15 correctly in ~2s, so it is now a rung in the ladder: media in hand is a 200, Hiker's own 404 is a definitive death certificate (404 not_found, no Apify wait), and a Hiker miss changes nothing.

• New rung after embed: Hiker v2 media/by/code (raw api/v1 envelope — v1's flattened shape made carousels look hollow)
• Hiker media → 200 via the same map_feed_post contract; false-404 case CjaD8JErmKH now returns the post in ~16s
• Hiker 404 → non-retryable 404 not_found, Apify skipped (skipped_hiker_not_found); dead posts stop 502ing as retryable
• stages.hikerStatus / ms.hiker in error traces; Decodo still runs first so private/deleted keep their precise labels

tiktok/popular-songs 503'd CREATIVE_CENTER_UNAVAILABLE again today. Two separate causes: TikTok's Creative Center login gate (up since Sep 4) still blocks the song chart on every path — native page render, direct rank_list API (40101), and the extended actor all hit the same wall, so the honest 503 stands. But the probe also caught a real regression: Decodo removed the legacy {type: wait, timeout: N} browser action, so the shared Creative Center xhr capture (hashtag / video / creator charts) failed validation with a 400 before ever reaching TikTok. The capture now sends the current wait_time_s schema. Failure streaks for the song chart also survive sparse probes: a 42h gap used to reset consecutiveFailures to 1 and re-open “Retry shortly” for a chart dead for 18 days.

• Creative Center xhr captures send {type: wait, wait_time_s: N} — the legacy timeout key 400'd at Decodo validation since ~mid-September
• popular-songs failure streak TTL 24h → 7 days: sparse daily probes keep consecutiveFailures and persistent=true honest; any 200 still clears the streak instantly
• song chart login gate re-verified 2026-09-22: anonymous page renders no chart (user/info answers 40000), direct rank_list answers 40101, extended actor returns 0 rows

search-companies → company-ads for 28 Day Lymphatic Reset Plan returned 20 near-identical inactive vagus-nerve variants (first row hollow, no hasMore) while Meta's live page was showing newer active creatives. company-ads always requested active_status=all, sliced the first HTML batch to limit, and omitted the search envelope — so 20 historical collation siblings looked like the whole library. Default is now ACTIVE (same as /facebook/search); status=ALL keeps the archive. nextCursor pages the fetched HTML batch; hasMore is true only when a cursor is present.

• status=ACTIVE (default) / INACTIVE / ALL — Meta URL + local filter, so leaked inactive collation siblings are dropped
• hasMore / nextCursor / retrievableCount / searchResultsCount match /facebook/search
• libraryUrl from search-companies (active_status=all) is rewritten to the requested filter

A nightly Turkey scrape 502'd fetch_empty after six successful native runs. The first /reels render came back empty and a shared 20s budget never tried again or reached /explore/reels; yesterday's grid had also aged out of the 6h last-good window. Each listing page now has its own timeout with one /reels retry, last-good lasts 26h (same as channel-reels), and the 502 body is flat (code/country/message) instead of error.error.

• Empty first /reels render retries once, then /explore/reels, instead of immediately 502ing
• Last-good snapshot is kept 26h so a once-a-night miss serves yesterday's grid as stale
• 502 envelope is {code, country, message, retryable} — no nested error.error

Nightly polls of @vogue and @theverge 502'd UPSTREAM_UNAVAILABLE / fetch_exhausted for a week. Those handles are no longer the magazine accounts: @vogue is a private stub (Vogue Covers, 0 followers, 1 post) and @theverge is a public profile with 0 posts. A live profile whose Reels tab is empty or locked is a successful read, not a server failure — channel-reels now returns 200 with an empty list, noPublicReels: true, and isPrivate when the account is locked. Deleted handles still 404 PROFILE_NOT_FOUND.

• Private profiles → 200, reels: [], noPublicReels: true, isPrivate: true
• Public profiles with zero posts or an actor no_items row → 200 empty, not a retryable 502
• Deleted / renamed handles stay 404 PROFILE_NOT_FOUND (retryable: false)

A live sweep of all Pinterest endpoints with valid and deliberately wrong inputs found two honesty gaps: a nonexistent pin id answered a billed 200 with an empty card, and pin URLs passed to profile/board endpoints were scraped for ~20 seconds as username 'pin' before failing. Valid pin-details, user-pins, user-boards, board and search paths (including cursor paging) already filled correctly.

• pinterest/pin-details: a pin id Pinterest does not have is now 404 Pin not found (was a billed 200 carrying only platform/id/url and a synthesized imageOriginal)
• pinterest/pin-details: a profile URL is 400 WRONG_RESOURCE_TYPE suggesting /v1/pinterest/user-pins (was a generic BAD_REQUEST)
• pinterest/user-pins + /board: a /pin/ URL is rejected instantly with 400 WRONG_RESOURCE_TYPE suggesting /v1/pinterest/pin-details (was ~20s of scraping username 'pin' ending in 404)

A live sweep of the Amazon seller storefront endpoint with valid and deliberately wrong inputs confirmed products, seller cards, cursor paging and metadata-only calls fill correctly, and dead sellers, influencer /shop/ URLs, product URLs and garbage are definitive 400/404s. One gap: an unknown marketplace code silently scraped amazon.com while the response stayed stamped with the junk code.

• amazon-shop/page: marketplace codes outside the supported set (US, UK, DE, FR, IT, ES, CA, JP, IN, AU, MX, BR) are now 400 INVALID_MARKETPLACE instead of silently answering US data stamped marketplace=ZZ

A live sweep of all Truth Social endpoints with valid and deliberately wrong inputs confirmed profile, post and cursor-paged user-posts fill correctly on the native path, dead handles and dead post ids are definitive 404s, and cross-platform URLs are rejected with WRONG_RESOURCE_TYPE. One gap: a Truth Social profile URL passed to /post answered a generic BAD_REQUEST.

• truth-social/post: a profile URL (truthsocial.com/@user) is now 400 WRONG_RESOURCE_TYPE with suggestedEndpoint /v1/truth-social/profile instead of a generic BAD_REQUEST
• URL classifier now recognises Truth Social posts and profiles, so detected{} blocks and endpoint suggestions include truthsocial.com URLs on every platform's endpoints

A live sweep of all Rumble endpoints with valid and deliberately wrong inputs found two honesty gaps: a channel that does not exist answered 200 with an empty video list, and passing a channel URL to a video endpoint (or vice versa) returned a generic BAD_REQUEST. Valid video-details, transcript, comments, channel-videos and search paths already filled correctly.

• rumble/channel-videos: a channel Rumble 404s is now 404 Channel not found (was a billed 200 with videos: [])
• rumble/comments: a video page Rumble 404s is now 404 Video not found instead of an empty 200
• rumble video endpoints: a Rumble channel URL is 400 WRONG_RESOURCE_TYPE suggesting /v1/rumble/channel-videos (and a video URL on channel-videos suggests /v1/rumble/video-details) instead of a generic BAD_REQUEST

A 27-call live sweep of the LinkedIn surface found four issues. post-details/post-transcript accepted /in/ and /company/ URLs, spent up to 70s scraping, then answered an all-null 200 built from the Apify actor's 'No post found or wrong input' stub — wrong resource types are now WRONG_RESOURCE_TYPE 400s before any scrape, and hollow actor stubs are 404s. Ad Library ad-details answered 503 retryable for ids that are not ads (LinkedIn serves a hydrated 'Ad Details' page without an ad body) — that page is now a definitive 404 that skips the Apify fallback. company-posts 502'd whenever native guest embeds came back empty: the 85s native budget left the Apify actor (needs ~60-80s) at most 17s, so every native miss was a guaranteed 502 — budgets are rebalanced (native 35s; an empty batch hands the actor the whole remaining window).

• linkedin/post-details + post-transcript: /in/ and /company/ URLs are WRONG_RESOURCE_TYPE 400s with suggestedEndpoint (were all-null 200s after a 70s scrape)
• linkedin/post-details + post-transcript: deleted/nonexistent posts are 404s (Apify 'No post found' stubs no longer normalize into null payloads)
• ad-library/linkedin/ad-details: ids that are not ads are a definitive 404 (hydrated detail page without an 'About the ad' body) instead of a 50s 503
• linkedin/company-posts: native budget 85s -> 35s and the Apify stage gets the full remaining window when nothing was collected — native misses no longer guarantee a 502

A 37-call live sweep of the YouTube surface found two honesty bugs. /comments answered 502 retryable on videos whose uploader turned comments off — it now returns 200 with commentsDisabled: true and an empty list. /shorts/* accepted any ≤3-minute long-form video because details were fetched through a pre-canonicalized /shorts/ URL, which stamped isShort on everything; the gate now asks YouTube directly (youtube.com/shorts/<id> 303-redirects to /watch for long-form) with no added latency, so 61–180s genuine Shorts shared as watch links are accepted and long-form pretenders are 422 before billing. All other valid and invalid-input paths (channel, videos, streams, playlists, community, search, hashtag, transcript, sponsors) already answered correctly.

• youtube/comments: uploader-disabled comments are a definitive 200 {commentsDisabled: true, comments: []} — no more 502 retryable loop
• youtube/comments: commentsDisabled: false is now always keyed on normal pages
• youtube/shorts/*: the Shorts gate is decided by YouTube's own /shorts/<id> routing, probed concurrently with details (no extra latency)
• youtube/shorts/*: 61–180s genuine Shorts pasted as watch?v= links are now accepted; ≤3-min long-form videos no longer masquerade as Shorts

A live sweep of Instagram social endpoints found two honesty bugs on /highlights: a YouTube URL 400'd as a missing url/userId instead of WRONG_RESOURCE_TYPE, and url=nope walked Decodo for 72s then answered 503 retryable. Valid channel, posts, details, comments preview, transcript, embed, hashtag and profile-search paths already answered correctly.

• instagram/highlights: a cross-platform URL is WRONG_RESOURCE_TYPE before handle extract (youtube.com/@nasa used to 400 as 'Provide url or userId')
• instagram/highlights: a handle Instagram positively does not have is 404 without the Decodo tray walk (url=nope was a 72s 503 retryable)

A live sweep of TikTok social endpoints with valid and deliberately wrong inputs found two honesty bugs: live?url=nope answered 200 isLive:false as if the account existed, and search-suggestions country=XX echoed the junk code on a real suggestion list. Valid channel, posts, video, comments, transcript, search and followers paths already filled correctly.

• tiktok/live + live-info: a handle TikTok positively does not have is 404 Creator not found (was a billed 200 isLive:false with an empty creator)
• tiktok/search-suggestions, popular-creators, popular-hashtags, trending-feed: non-ISO country codes are 400 INVALID_COUNTRY instead of a global result stamped country=XX
• tiktok/song-details: a profile URL is WRONG_RESOURCE_TYPE (expected music) instead of a generic BAD_REQUEST

A live sweep of every Facebook Ad Library endpoint with valid and deliberately wrong inputs found two honesty bugs: junk ad ids and page URLs answered 503 retryable in under a second, and a garbage company-ads url silently became a keyword search that returned 20 unrelated ads. Valid search, company-ads, ad-details and transcript paths already filled to the requested limit.

• ad-library/facebook/ad-details + ad-transcript: garbage ('nope'), short ids ('1') and facebook.com page URLs are a definitive 400 before any fetch (was 503 upstream_unavailable / retry shortly)
• ad-library/facebook/company-ads: inputs that are not a pageId, libraryUrl with view_all_page_id, or a facebook.com page URL are 400 BAD_REQUEST (url='not a page' used to return 20 keyword-search ads)
• ISO country, q length, status/media/date/limit validation already 400/422 as documented; cross-platform URLs stay WRONG_RESOURCE_TYPE

Logged-out Instagram still cannot page a 500-comment thread or resolve highlight albums from profile-HTML ids. Two caller-facing lies around that limit are fixed: asking for the 2-comment preview no longer 503s, and an ids-only highlight shelf is no longer a retryable 503.

• instagram/comments: limit<=2 returns the logged-out Polaroid as 200 with truncatedReason (was 503 SESSION_UNAVAILABLE after a session walk when Instagram embedded 1 of 509)
• instagram/comments: limit>2 of a larger thread is still 503 SESSION_UNAVAILABLE at 0 credits — that window still needs a live session
• instagram/highlights: HTML-only ids that do not resolve on highlights-details are 422 ids_unresolvable, retryable:false (was 503 retry-in-30s). Retrying does not make those ids work

A live sweep of every Reddit and Twitter endpoint with valid listings plus deliberately wrong inputs. Feeds, search, pagination and most validation were already honest. Three caller-facing bugs are fixed: common share URLs were 400, and a junk community id answered 200 with an empty card after a 21s Apify walk.

• reddit/post-details, post-comments, post-transcript: redd.it/ID and reddit.com/comments/ID are accepted (were 400 Invalid Reddit URL)
• twitter/tweet-details and transcript: x.com/i/web/status/ID and x.com/i/status/ID share forms are accepted (were 400)
• twitter/community: ids under 10 digits are 400 before scrape; a nameless Apify stub is 404 Community not found (was 200 {platform,id,url} in 21s)
• twitter/community-tweets: a tweet/status URL is WRONG_RESOURCE_TYPE pointing at tweet-details

A live sweep of Threads and Spotify with valid listings plus deliberately wrong inputs. Spotify was already honest (wrong-kind URLs 400, dead artist 404, search fills to the requested limit). Threads post-details 404'd a URL we had just listed — Meta no longer puts the target post inside thread_items — and garbage handles were scraped then 404'd instead of rejected.

• threads/post-details: permalink HTML still embeds the post; the parser now walks out from "code" past nested caption objects instead of requiring thread_items (those arrays are suggested posts)
• threads/profile and user-posts: handles with spaces or other illegal characters are 400 INVALID_HANDLE before any scrape (was 404 after a 4s fetch)
• Spotify artist/track/album/podcast/search: no fill or validation bugs on this sweep

A live sweep of TikTok Shop, TikTok/LinkedIn/Google Ad Library found two places where we already had the ads and then threw them away, plus a shop catalog URL that silently returned zero products. Literal keyword hits still lead; the rest of the already-fetched page fills behind them up to limit.

• ad-library/tiktok/search: literal hits first, already-fetched SERP ads fill up to limit (rankedFill + matchedFrom [])
• ad-library/linkedin/search-ads: keyword no longer drops scraped cards — they fill the limit behind literal hits
• tiktok-shop/shop-products: slug-only /shop/store/{name} is a definitive 400 (needs numeric shop id)

A live sweep of the TikTok Shop and TikTok/LinkedIn/Google ad library endpoints with deliberately wrong inputs found several cases where a definitive answer was disguised as a retryable 503. Limit fill was also audited: every listing endpoint returns exactly the requested count when upstream has the ads.

• ad-library/tiktok/ad-details: an id TikTok's library positively reports as nonexistent is now 404 AD_NOT_FOUND (was a 42s fallback walk ending in 503 retryable)
• ad-library/google/ad-details: a creative absent under the advertiser is 404 AD_NOT_FOUND when upstream answered (503 stays reserved for transport failures)
• ad-library/google company-ads + advertiser-search: unknown brands return the documented resolution: not-in-library 200 at 0 credits (Google's empty-suggestions reply was misread as an outage → 503)
• ad-library tiktok/linkedin ad-details: garbage ids like 'notanumber' are a definitive 400 before any fetch
• ISO 3166 country validation extended to linkedin/search-ads (each code in countries), google company-ads/ad-details/advertiser-search and tiktok/top-ads — non-ISO codes are 400 INVALID_COUNTRY instead of silently empty/global results
• ad-library/google/company-ads: limits above 40 now page Google's RPC until the limit is met (limit=50 used to return 40 with hasMore:true)
• ad-library/linkedin/search-ads: limits above one SERP page (~25 cards) pull further pages automatically within the time budget, and the returned paginationToken always follows the last fetched page

A live sweep of all 20 Facebook endpoints with deliberately wrong inputs surfaced four behaviour bugs, all fixed. Caller errors that used to look transient (retryable 502s, silently 'working' filters) are now definitive 400s with actionable messages, and profile-reels no longer declares NO_PUBLIC_REELS off a single degraded render.

• facebook/group-posts with a profile/page URL: definitive 400 pointing to profile-posts (was 502 retryable:true)
• facebook/marketplace-search with an unresolvable place name: 400 UNKNOWN_LOCATION before any scrape (was 200 with listings from an unrelated metro)
• ad-library/facebook search, company-ads, search-companies: non-ISO country codes are 400 INVALID_COUNTRY (Meta used to silently search globally while echoing the junk code)
• facebook/profile-reels: a second confirmation render precedes the NO_PUBLIC_REELS verdict — degraded snapshots no longer masquerade as 'no public Reels'
• facebook/comments: a page/profile URL returning the newest post's comments is now documented

Numeric profiles (facebook.com/61593…) were timing out at 90s (503) or failing at ~85s (502) — the /reels and /videos tabs were rendered one after another, and per-reel hydration had no overall deadline, so a slow batch was cancelled wholesale at the route ceiling. The two listing tabs now race in parallel, hydration keeps every reel that finished inside the budget instead of discarding them, and reel permalinks on numeric profiles (which have no vanity slug) are now mined too.

• /reels and /videos listing renders race in parallel — failures answer in ~45s instead of ~85s
• per-reel hydration keeps finished rows at the deadline instead of a 503 with nothing
• numeric-profile /videos permalinks (no vanity slug) now yield reel ids

facebook.com/share/r/… (mobile Share button) was a 400 UNRESOLVED_SHARE_LINK asking you to open the link in a browser — logged-out Facebook stopped answering the redirect we relied on. Share links (/share/, /share/p|r|v/) now resolve via a headless render that reads Facebook's own og:url, cached 30 days per share token. The /share/r/<id> form also no longer mis-parses its id as 'r'.

• share/p|r|v links resolve to the canonical post/Reel automatically
• share→canonical mapping cached 30 days — repeat pastes are instant
• slug-form /videos/<seo-slug>/<id>/ URLs classify as video, not page

An extended / Apify 200 for a URL-only channel-reels call returned 15 complete reels but a hollow top-level user{} (id, displayName, userId all null) even though every reel author carried the numeric id and display name. Handle→ID resolve had missed, so the envelope kept a username stub and never copied the matching author. user.id / userId / displayName now lift from authors whose username matches the queried handle — never from a collab first row. followers / verified / avatar stay null when Instagram omitted them. hasMore stays null on degraded (unknown), not a false end.

• user.id / userId / displayName filled from matching reel authors on the extended path
• native and extended share one envelope key set (user{}, userId, partial, degraded)
• collab first rows (herbalife on cristiano) still do not become user{}

A same-day burst of /instagram/details 502s (one user, six shortcodes, /p/ then /reel/ 150ms later) was not a global Instagram outage — 108 other details calls were native 200s. Logged-out GraphQL could sit until the 15s wrapper cancelled the session hop, views backfill could throw away a mapped post, /p/ fail-cache replayed /reel/ without fetching it, and APIFY_BLOCK_ENDPOINTS still disabled the actor on details. Logged-out is now 8s, session always gets 6s, a mapped post is kept, Decodo tries the other permalink, and details ignores the Apify kill switch the same way listing already did.

• logged-out GraphQL hard-capped at 8s so the cookie hop still runs
• /p/ and /reel/ both fetched before the 60s fail-cache locks the shortcode
• APIFY_BLOCK_ENDPOINTS cannot disable /instagram/details

region=BR search returned www.tiktok.com/shop/pdp/{id} with integer centavos and no currency. Pasting that URL into product-details defaulted to US and 404ed a live BR listing. Search now emits shop.tiktok.com/{cc}/pdp/{id}, hydrates that storefront, fills currency from the market, and converts integer minor units (1282 → 12.82 BRL). A full extended page sets hasMore=true instead of pretending the catalog is exhausted.

• BR/MX/… search URLs are shop.tiktok.com/{cc}/pdp/{id}
• SERP hydrate fetches the market PDP, not the US www path
• missing currency filled from region; integer tiles become major units
• extended window hasMore=true when the page is full

When native Creative Center missed, the parked Apify fallback still started with apifyProxyGroups:[SHADER]. This account does not have datacenter proxies, so Apify returned 400 invalid-input and we surfaced that as a 503 after ~56s — then retried the same broken payload. The actor now uses RESIDENTIAL (same group as the other TikTok actors). A proxy-group denial retries once without a named group; other invalid-input 400s are not retried.

• Top Ads Apify input defaults to RESIDENTIAL, not SHADER
• SHADER / proxy-group 400 retries once without apifyProxyGroups
• other invalid-input 400s fail immediately (no second hop)

q=healthy drink scanned 15 Creative Center rows and returned ads=[] because none of the titles had those whole words. TikTok had already ranked that page for the keyword — the same list the website shows. Those rows now come back as matchBasis=ranked with literalMatches=0 (no matchedFrom). Whole-word hits still filter as before. orderBy=like echoes likes, matching the documented enum.

• no whole-word hit → ranked page (matchBasis=ranked), not ads=[]
• Spark author nickname is in the literal haystack (same as brandName)
• orderBy echoes likes / impressions, not CC's like / impression

www.tiktok.com/shop/pdp/{id} is the canonical PDP URL — we already parse it. A live BR listing (shop.tiktok.com/br/pdp/{id}, priced in R$) still 404ed because the default region is US and the path has no market. The endpoint now reads /{cc}/pdp/ from shop.tiktok.com and, after a US-default miss, probes other Shop storefronts in parallel before declaring the product gone.

• shop.tiktok.com/br/pdp/{id} sets region=BR from the path
• US-default miss probes BR/MX/GB/ID/TH/VN before 404
• resolved market is echoed as data.region

Healthy Marketplace searches return in ~28s. A hung Decodo first-paint sat through 55s + 20s drain and came back as 504 UPSTREAM_TIMEOUT at exactly ~75s; an empty login shell was 502 UPSTREAM_UNAVAILABLE with no second chance. Search now hedges like marketplace-item: a fresh session starts at 32s and the first HTML that carries listing cards wins. A fast empty/transport miss retries once. resourceUrl is the Facebook search URL instead of null.

• hedged second Decodo session at 32s — first usable listings win
• empty login-shell / transport miss retries once instead of 502
• resourceUrl is the Marketplace search URL, not null

Country-localized /reels listing found shortcodes (Turkey, and the same class of JP/KR misses) but Polarıs hydrate ran from generic US residential and returned zero posts. In-flight GraphQL calls had no per-item cap, so 24 codes walked past the 110s router deadline. Listing HTML media is mapped first; remaining codes hydrate through a country-pinned residential hop with an 8s cap; a last Decodo permalink pass uses the same geo. Failures still 502 with no cached lie — they just stop hanging.

• listing HTML media hydrates the grid without a Polarıs round-trip
• Polarıs residential egress is pinned to the requested country (TR, JP, …)
• each Polarıs item is wait_for-capped at 8s; Decodo geo permalink fills a Polarıs miss

The endpoint treated user.ttSeller=false as “no shop” and returned products:[] without fetching the showcase (showcaseMs=0). Affiliate creators are ttSeller:false by definition, so every affiliate lookup was a billed empty array. The shelf is always fetched now. no_shop means we looked and found nothing; has_products includes affiliate creators. Looked-and-empty stays 1 credit; products bill per result.

• affiliate (ttSeller:false) showcases are fetched instead of short-circuited
• sellerStatus no_shop is a looked-and-empty result, not a skipped fetch
• docs + FAQ no longer say ttSeller:false skips Apify

advertiser-search no longer returns a single Italian NIKE SRL for q=nike&country=US. Brand queries are expanded (Nike, Inc. + siblings), ranked with country preference (US prefers Inc. over SRL/BV), and return multiple AR entities so you can pick. company-ads adds resolvedAdvertiser {id,name,url}, sort=last_shown|first_shown, and stable null keys for text/headline/cta/landingUrl/spend. Chain: search -> advertisers[0].id -> company-ads?advertiser=AR...

• advertiser-search: expand + rank multi-result (Nike, Inc. first for US)
• company-ads: resolvedAdvertiser + sort + stable null schema
• Docs/examples use AR167 Nike, Inc. for the creatives chain

Post analytics no longer ships views-only YouTube rows when watch-page enrich is thin - incomplete native responses fall through to Apify so likes, comments, publishedAt, and @handle populate on the vitrin example. author.username never echoes the display name. metrics.engagementRateBasis is always interactions/views on post/compare (do not compare to popular-creators). Compare adds status, failedCount, and failed[] so partial batches are auditable; publishedAt normalizes to UTC Z; docs never fall back to example.com lorem.

• YouTube thin native -> Apify when likes/comments/handle/date missing
• username = @handle only; displayName stays separate
• engagementRateBasis=interactions/views on every post metrics object
• compare: status + failed[] + failedCount; real docs example fallback

Compare analytics docs no longer show placeholder example.com rows. The live example returns count/resolved/results[] with the same metrics object as post analytics (views, likes, comments, engagementRate, …). Adds cache=true (per-URL cache shared with /post; hits free) and honest billing copy — 1 credit per resolved URL, no bulk discount.

• Live snapshot example with two real YouTube URLs and full metrics{}
• Additive cache param (shared with /v1/analytics/post)
• Docs/FAQ: same shape as post analytics; no bulk credit discount

Cross-platform post analytics now uses the same enriched YouTube path as video-details, so likes, comments, publishedAt, and engagementRate populate instead of staying null when views alone were present. author.username prefers the channel handle; shares/saves remain null on YouTube (not publicly exposed).

• YouTube analytics reuses enriched video-details (likes/comments/publishedAt)
• author.username ← channelHandle when available; displayName stays channel name
• Docs/FAQ: same metrics shape; platform-missing fields stay null

Reddit post-comments publishedAt is now ISO 8601 (was a stringified unix float like 1785330725.0, which broke Date parsers). Existing flat comments + depth/parentId stay; response adds post, score/downs, authorFullname, and hasMore.

• publishedAt is ISO 8601 UTC
• Additive score, downs, authorFullname, distinguished, controversiality
• post object + hasMore/nextCursor (cursor paging still deferred)

LinkedIn profile no longer copies og:description into about, and no longer invents connections/currentCompany from that SEO trailer (the Bill Gates 8 connections bug). about prefers JSON-LD Person description and is omitted when only SEO meta is available; connections matching the privacy/SEO placeholder are null.

• about from JSON-LD only - SEO meta never returned as about
• connections matching SEO N connections on LinkedIn chrome -> null
• Additive experience[] / education[] when the Apify fallback returns them

Whisper-based transcription no longer invents text for videos that contain only music or silence.

• Known hallucination phrases, subtitle credits, and music annotations are filtered out
• Low-confidence segments are dropped; music-only clips now return 422 No speech instead of made-up text
• Wrong-language output (e.g. Turkish speech labeled as Russian) fixed via validated retries with a pinned language