{"id":17460,"date":"2020-11-19T15:00:53","date_gmt":"2020-11-19T15:00:53","guid":{"rendered":"http:\/\/www.wjst.de\/blog\/?p=17460"},"modified":"2021-02-09T14:17:45","modified_gmt":"2021-02-09T14:17:45","slug":"how-to-scrape-a-website-with-r-i-using-a-browser-generated-cookie","status":"publish","type":"post","link":"https:\/\/www.wjst.de\/blog\/sciencesurf\/2020\/11\/how-to-scrape-a-website-with-r-i-using-a-browser-generated-cookie\/","title":{"rendered":"How to scrape a website with R I: Using a browser generated cookie"},"content":{"rendered":"

While there are quite some SO examples<\/a> out there how to manage the login, here are the ncessary steps whenever you need to login in manually and have to start with a browser cookie. First install the “EditThisCookie<\/a>” plugin in Chrome and export the cookie<\/p>\n

to the clipboard.<\/p>\n

\"\"<\/p>\n

Then cut and paste the clipboard content into the R editor in the 3rd line<\/p>\n

\r\ninstall.packages(c("rvest","stringr","rjson","RCurl")\r\nc.json <- c('\r\n[\r\n{\r\n    "domain": "www.domain.de",\r\n    "hostOnly": true,\r\n    "httpOnly": true,\r\n    "name": "MX_SID",\r\n    "path": "\/",\r\n    "sameSite": "unspecified",\r\n    "secure": true,\r\n    "session": true,\r\n    "storeId": "0",\r\n    "value": "4e1799f010efa4387874399253695521",\r\n    "id": 1\r\n}\r\n]\r\n')\r\nresult <- fromJSON(c.json)\r\nfn <- c("\u02dc\/cookies.txt")\r\nunlink(fn)\r\ne <- if (!exists("i$expirationDate")) 2147483647 else 0\r\nfor (i in result) {\r\n  cat( paste0(i$domain,"\\t",\r\n    "FALSE","\\t",\r\n    i$path,"\\t",\r\n    "TRUE","\\t",\r\n    e,"\\t",\r\n    i$name,"\\t",\r\n    i$value ), file=fn, sep="\\n", append=TRUE )\r\n}\r\ncurl = getCurlHandle (cookiefile = fn,\r\n    cookiejar = fn,\r\n    useragent = "Mozilla\/5.0 (Windows; U; Windows NT 5.1; en - US; rv:1.8.1.6) Gecko\/20070725 Firefox\/2.0.0.6",\r\n    verbose = TRUE)\r\ncontent <- getURL("https:\/\/www.domain.de\/whatever",curl = curl)\r\nrm(curl)\r\n<\/pre>\n\n
 <\/p>\n
\n  CC-BY-NC Science Surf , accessed 24.04.2026<\/span>\n <\/div>","protected":false},"excerpt":{"rendered":"
While there are quite some SO examples out there how to manage the login, here are the ncessary steps whenever you need to login in manually and have to start with a browser cookie. First install the “EditThisCookie” plugin in Chrome and export the cookie   CC-BY-NC Science Surf , accessed 24.04.2026<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9],"tags":[2893,3573,2847,3572,3570,3568,3569,3571],"class_list":["post-17460","post","type-post","status-publish","format-standard","hentry","category-computer-software","tag-r","tag-rcurl","tag-cookie","tag-rjson","tag-rvest","tag-scraper","tag-ssl","tag-stringr"],"_links":{"self":[{"href":"https:\/\/www.wjst.de\/blog\/wp-json\/wp\/v2\/posts\/17460","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.wjst.de\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.wjst.de\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.wjst.de\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.wjst.de\/blog\/wp-json\/wp\/v2\/comments?post=17460"}],"version-history":[{"count":8,"href":"https:\/\/www.wjst.de\/blog\/wp-json\/wp\/v2\/posts\/17460\/revisions"}],"predecessor-version":[{"id":18060,"href":"https:\/\/www.wjst.de\/blog\/wp-json\/wp\/v2\/posts\/17460\/revisions\/18060"}],"wp:attachment":[{"href":"https:\/\/www.wjst.de\/blog\/wp-json\/wp\/v2\/media?parent=17460"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.wjst.de\/blog\/wp-json\/wp\/v2\/categories?post=17460"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.wjst.de\/blog\/wp-json\/wp\/v2\/tags?post=17460"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}