A place for developers to talk about building on Farcaster.

/dev

what's the best way to programmatically scrape an entire twitter profile? 

i would pay to do it rather than set it up myself if its not overpriced

Opacity

product @seedclub. i host S-TIER @ X Spaces & i interview crypto builders @ web3galaxybrain.com.

vibes and status at /opacity --
host of vibe check podcast 
▶️ https://pods.media/vibe-check
❄️ icebreaker.xyz/dawufi
👀 seemore.tv/free/fc/dawufi

i want to enable people to scrape their own -- or do so on their behalf after they twitter auth

Your own or someone else's? We can help depending on what you're trying to gather up (we being /opacity)

I've actually done this flow once before. 
Clerk for auth, Third Party API (Apify) for scraping tweets.
But the api can accrue cost quickly depending upon how many items you want

Without relying on third party APIs, you can leverage a web automation framework like Puppeteer to scrape everything off of a Twitter profile. Dm me and I’ll get you set up