this article is mainly centered around scraping in javascript (node.js, bun, deno)
sometimes, you need to programmatically access a website's data. many websites provide a public api, and depending on the website, it can suffice. but in many cases, they're limited, paid, and/or slow
an alternative a lot of newbies aren't aware of is scraping. scraping is the act of accessing data on a website without using an intentional means of doing so
over the years, i've gained a lot of knowledge about doing this. it's one of my favorite parts of coding in general. this post will cover a lot of the methods i use to scrape websites
puppeteer? not where we're going
one of my biggest gripes is people who use puppeteer to scrape websites. 99% of the time, you don't need a full browser running in the background to effectively scrape. it's often MUCH faster to pretend to be a browser, and much more memory-affordable
sometimes, it's justified. for example, there are anti-bot solutions like twitter's x-client-transaction-id header, where you just need a browser to do the challenge. that's fine, but before you take that route, you should look for everything i'm describing in this post
the first step
open your browser's devtools, go to the network tab, and refresh a website. you'll see a lot going on. the first step to scraping is mimicking the request that returns the information you want
first of all, i like to use firefox for this. it has no Sec-Ch-* headers to worry about, and it's a fairly common browser across all operating systems
now you have to find the request you're looking for. some websites embed everything directly in the html response, some return a template and fill it in with a private api (often graphql). if you can, the private api is usually the way to go, since it doesn't involve parsing and if a website has one then it's probably not an easily scrapable document. it really depends on the website though
click through the requests and search keywords you're looking for in the responses. you'll find what you're looking for pretty fast
scraping documents
for html, i like to use jsdom, but it's kind of slow. if speed is critical, i'd recommend using node-html-parser. it's not fully standards compliant, but it works fine
so now, to get the values you want, you have to find where they actually are in the document. websites using vanilla html and no crazy frameworks are usually formatted in a way where this is very easy. let's use bandcamp as an example
in this page, i want to retrieve the name of the album. on bandcamp, all of the common methods i use to scrape from a document are used, so it's a very good example
method 1: document
on the page, there is large text saying the name of the album. you can right click that, click inspect element, and it'll select the element you right clicked
in this case, it's an h2 with a class of "trackTitle", here's how i'd extract that info:
let albumName = document.querySelector('.trackTitle').textContent.trim()
console.log(albumName) //=> Sitting At The End Of The World
i trim it because it has a lot of spaces and linebreaks due to formatting. this doesn't show up in the actual rendered page due to how html is rendered
the downside to this is that if they change their layout, you usually have to rewrite it. this doesn't always apply to the next method though
method 2: metadata
many websites return metadata about the page in meta tags. these are usually a good source of clean data, and depending on the website, usually better to use than the document itself (especially if it's one of those modern frontends with complicated html)
when doing this, i'd recommend using a common metadata crawler's user agent for a lesser chance of being blocked, i typically use the following:
facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)then you search for the keyword you're looking for in the html, and find the meta tag it's in
on this page, what i'm looking for is a meta tag with the name "title", here's how i'd extract that info:
let albumName = document.querySelector('meta[name="title"]').getAttribute('content')
//remove the ", by JOHNNASCUS" part, which is present in the meta tag for the title on bandcamp
let parts = albumName.split(',')
parts.pop()
albumName = parts.join(',')
console.log(albumName) //=> Sitting At The End Of The World
method 3: embedded json
sometimes, a website will have a string of json containing the data you're looking for. if you can do it like this, this is the best way to scrape from a document as it has the cleanest data and requires very little bullshit
the easiest way to find this is to just search the keyword in the html, and see if it appears in a script, or in an attribute
on this page, what i'm looking for is a script with the field "data-tralbum"
let script = document.querySelector('script[data-tralbum]')
let tralbum = JSON.parse(script.getAttribute('data-tralbum'))
let albumName = tralbum.current.title;
console.log(albumName) //=> Sitting At The End Of The World
using private apis
sometimes, the document is too hard, or even nearly impossible to parse. a good example of this is Astronomy Picture of the Day, which is an older website with older conventions
in this case, you'll want to try to find a private api. there are many places you can look, but the first place you should look is your devtools
the only thing you have to worry about here is authorization, usually some kind of static anonymous authorization or a client id in a script. if the auth is static, just copy that and you're good. if it's dynamic, find out where it comes from and make that same request. if it's in a script, use a regex to parse it out
i sadly don't have any examples for this particular site, or a site containing all of these, so i'll just give you some general info
graphql hell
this one is the worst. i fucking hate graphql. lets just get it out of the way
the best example of this is twitter, which uses graphql in it's web app. the endpoints are insane:
https://x.com/i/api/graphql/jbkn58DkVk8bIkDp8IEAsQ/TweetDetail?variables={"focalTweetId":"1734619125081903291","with_rux_injections":false,"rankingMode":"Relevance","includePromotedContent":true,"withCommunity":true,"withQuickPromoteEligibilityTweetFields":true,"withBirdwatchNotes":true,"withVoice":true}&features={"rweb_video_screen_enabled":false,"profile_label_improvements_pcf_label_in_post_enabled":true,"rweb_tipjar_consumption_enabled":true,"responsive_web_graphql_exclude_directive_enabled":true,"verified_phone_label_enabled":false,"creator_subscriptions_tweet_preview_api_enabled":true,"responsive_web_graphql_timeline_navigation_enabled":true,"responsive_web_graphql_skip_user_profile_image_extensions_enabled":false,"premium_content_api_read_enabled":false,"communities_web_enable_tweet_community_results_fetch":true,"c9s_tweet_anatomy_moderator_badge_enabled":true,"responsive_web_grok_analyze_button_fetch_trends_enabled":false,"responsive_web_grok_analyze_post_followups_enabled":true,"responsive_web_jetfuel_frame":false,"responsive_web_grok_share_attachment_enabled":true,"articles_preview_enabled":true,"responsive_web_edit_tweet_api_enabled":true,"graphql_is_translatable_rweb_tweet_is_translatable_enabled":true,"view_counts_everywhere_api_enabled":true,"longform_notetweets_consumption_enabled":true,"responsive_web_twitter_article_tweet_consumption_enabled":true,"tweet_awards_web_tipping_enabled":false,"responsive_web_grok_show_grok_translated_post":false,"responsive_web_grok_analysis_button_from_backend":true,"creator_subscriptions_quote_tweet_preview_enabled":false,"freedom_of_speech_not_reach_fetch_enabled":true,"standardized_nudges_misinfo":true,"tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled":true,"longform_notetweets_rich_text_read_enabled":true,"longform_notetweets_inline_media_enabled":true,"responsive_web_grok_image_annotation_enabled":true,"responsive_web_enhance_cards_enabled":false}&fieldToggles={"withArticleRichContentState":true,"withArticlePlainText":false,"withGrokAnalyze":false,"withDisallowedReplyControls":false}
and the responses are even more insane, i was going to put one here but it takes up 20x the post's length, so i'll link to the file
if you don't believe it's that bad, just look at this
to be fair, you have to note that twitter heavily misuses graphql. though, in other sites where it's used, it's also pretty hellish. just not like this LOL
the best way to go about it is to use something like jsonpathfinder and click shit until you find what you want. for the variables and shit in the url, just copy them and you'll be good (until they update the layout of the site or the endpoint)
you also almost ALWAYS need authorization on these, i don't think i've found one where you don't need some kind of authorization, even if signed out
general private apis
these are self explanatory, they're usually just clean json, easy to parse, won't have a problem
mobile apis
this is the good stuff. these are especially nice for websites that don't have a private api on the frontend, but do have a mobile app
it's a bit more involved, but it usually ends up being easier and safer than any other option, since these apis are generally less resistant to scraping
to actually find these apis, i recommend using mitmproxy, and proxying your phone. on ios, this is pretty straightforward. on android, you have to patch the app most of the time, so you should just use ios here if you can
once you're proxying it, open the app and do something that returns the data you want, and you'll see it pop up in the request log. the requests are usually much more minimal than the ones taking place in browsers
you're especially likely to have to worry about authorization in this scenario, since you usually have to log in on these mobile apps. if it's a different kind of auth than the website uses, you're kinda fucked, otherwise it's fine
these apis are usually pretty nice, and less limited since they're intended for the mobile app. it's especially freeing when a website that only returns compiled html has these nice json apis that are hidden in the mobile app
public endpoints
okay this doesn't exactly fit into "private apis", but i thought it's worth mentioning
sometimes, websites will have public endpoints for rss readers and things like that, these are really nice to have for websites that have hellish private apis all around
a good example of this is youtube's channel rss feed, it's literally made to be polled so there isn't even ratelimiting and it just... Exists, without auth or anything
these aren't exactly abundant and they sometimes don't provide all the info you need, but for rare cases like this, it's really nice
the only downside is that they're often, as i said, for things like rss readers, so the format usually isn't just json. for example, rss uses xml, so you have to use an xml parser to parse the contents of an rss feed endpoint
websites' protections against scraping
sometimes websites know what you're trying to do, this happened to me recently with spotify. they restricted their /get_access_token endpoint to use totp
in this case, i opened up all of spotify's js files, searched for a keyword, beautified the one that had it, and found out exactly what they're doing. in most modern sites, it's a lot of nonsense of single letter variable names and unreadable code, but you can figure it out
sometimes, the way it's implemented, it's just outright stupid. for example, there's a company called datadome, and one thing they do is provide some kind of anti-scraping service, which sends a lot of details about your browser to the server, and in return, the server sends you an access cookie back. all you have to do is copy the request, change the ones related to time, and it's good
other times, such as captchas, are more complex. there are such things as captcha solvers, usually paid services for advanced bots or actual human workers that get paid per captcha, but i don't really like to use them
in such situations, you should just avoid the captcha. sometimes just being even more like a browser will succeed in this, such as abiding by Set-Cookie and other stuff. if it's an automated captcha where the user doesn't even have to interact with it, you can usually just use a browser to solve it, then save that cookie and continue using that
one particular service of note is amazon, which has a terrible captcha. you can completely evade it by adding a random value to your user agent (like the current unix epoch). this is seriously enough to stop them lol
as well as all of that, some websites tend to like to limit you to only residential ip addresses. in other words, your scraping works on your network, but not your server's. the best way to combat this is using proxies, either using your home network as a proxy, or buying some residential proxies. just don't get too careless, because going crazy with proxies may step over the line of legality
one last thing, both twitter and amazon actually block node.js's default tls ciphers and you have to randomize them in order to be able to properly access them from it. these are the only two sites i've found that do this. courtesy of cobalt.tools, here's how you can get past that:
const tls = require('tls')
const crypto = require('crypto')
const ORIGINAL_CIPHERS = tls.DEFAULT_CIPHERS;
function randomizeCiphers() {
do {
let cipherList = ORIGINAL_CIPHERS.split(':')
let sliced = cipherList.slice(0, 8)
let shuffled = sliced;
for (let i = shuffled.length - 1; i > 0; i--) {
let j = crypto.randomBytes(4).readUint32LE() % shuffled.length; //fun javascript fact: if you remove the semicolon, EVERYTHING BREAKS
[ shuffled[i], shuffled[j] ] = [ shuffled[j], shuffled[i] ]
}
let retained = cipherList.slice(8)
tls.DEFAULT_CIPHERS = [ ...shuffled, ...retained ].join(':')
} while (tls.DEFAULT_CIPHERS === ORIGINAL_CIPHERS)
}
randomizeCiphers()
setInterval(randomizeCiphers, 1800000)
legality of scraping
i don't actually know if it's "legal" to scrape websites, but the general consensus is that if you're not trying to access content you otherwise shouldn't be able to, it's fine. there is a fine line between scraping and hacking, and if you're hacking, you should probably report that and get tons of moners...
regardless, i don't really care either. the aforementioned spotify endpoint has a key in it called "_notes", with the value "Usage of this endpoint is not permitted under the Spotify Developer Terms and Developer Policy, and applicable law". yet, i continue to scrape it because what are they gonna do? sue me for "GET /"? maybe...
on the contrary, bandcamp's track info field has a key called "for the curious", with a link to here, which basically says "we're based as fuck"
so there's no actual definitive proof on whether scraping itself is legal or not. if you're really worried, i'd take a look at hiQ Labs, Inc. v. LinkedIn Corp.
conclusion
to me, scraping is an art. to know how to effectively do it is a skill, but in realness, most websites don't even try to stop you
there's no real anti-scraping solution that really works. there's ALWAYS a way around, be that using a browser to solve challenges, actual human workers that solve captchas for you, residential proxies, and many others
despite the fact that i named my last post "the art of detection", i think this one deserved a similar name. it's a really fun part of coding for me
if you use this post to facilitate thousands of bots scraping a big website, PLEASE don't reference me in court. i'm just the messenger