{"id":2211,"date":"2015-07-24T09:18:22","date_gmt":"2015-07-24T08:18:22","guid":{"rendered":"http:\/\/www.matthijskamstra.nl\/blog\/?p=2211"},"modified":"2015-08-22T20:04:46","modified_gmt":"2015-08-22T19:04:46","slug":"scraping-with-haxe","status":"publish","type":"post","link":"https:\/\/www.matthijskamstra.nl\/blog\/2015\/07\/24\/scraping-with-haxe\/","title":{"rendered":"Scraping with Haxe"},"content":{"rendered":"

A quick post about something that grabbed my attention quickly.<\/p>\n

Scraping<\/h3>\n
\n

Web scraping (web harvesting or web data extraction) is a computer software technique of extracting information from websites.<\/p>\n<\/blockquote>\n

Source: https:\/\/en.wikipedia.org\/wiki\/Web_scraping<\/a><\/p>\n

I already did some research on the subject when I was playing around with my raspberry pi. There is a lot out there, especially for python.<\/p>\n

But what about my favourite programming language Haxe<\/strong>?<\/p>\n

Again this is a quick search! And this is what I found.<\/p>\n

A (very?) old project from Jonas Malaco Filho<\/a> on github. Check out this code : jonas-haxe<\/a> and specificly the scraper part<\/a> of it. Written for Neko, with primarily undocumented classes like neko.vm.Mutex<\/a> Once you have the html page you can start getting the data from it!<\/p>\n

You will need a html\/xml parser; I found one written by Yaroslav Sivakov<\/a> – HtmlParser haxe library<\/a> It also can be found on haxelib: http:\/\/lib.haxe.org\/p\/HtmlParser\/<\/a><\/p>\n

I found a little (old) project haxe\/php project that I will post as a reference https:\/\/github.com\/andor44\/old_scraper<\/a>. But then it stops…<\/p>\n

Not a field that a lot of haxe-developers walk. Fun!<\/p>\n

Update #2<\/h2>\n
    \n
  1. The htmlparser doesn’t work with the html code I am scraping. So I need to focus the parts I want to use. Regular expressions are the way to go, and I suck at them. Luckily I found a online tool that helps with testing the regex<\/a>: http:\/\/www.regexr.com\/<\/a> from an old flash hero gskinner<\/a>.<\/li>\n
  2. Another thing I ran into, was the data from https<\/strong> sites. You need something “extra” to download html files from there: install hxssl<\/a> via haxelib haxelib install hxssl<\/code> and add it to your build.hxml -lib hxssl<\/code><\/li>\n<\/ol>\n

    Update #1<\/h2>\n

    I am coding this with openfl\/regular expressions, but perhaps a better way to-go is node.js! And you can use node.js with Haxe (perhaps not completely ready: hxnodejs<\/a> but probably good enough for the examples below).<\/p>\n