Scraping with Haxe

A quick post about something that grabbed my attention quickly.

Scraping

Web scraping (web harvesting or web data extraction) is a computer software technique of extracting information from websites.

Source: https://en.wikipedia.org/wiki/Web_scraping

I already did some research on the subject when I was playing around with my raspberry pi. There is a lot out there, especially for python.

But what about my favourite programming language Haxe?

Again this is a quick search! And this is what I found.

A (very?) old project from Jonas Malaco Filho on github. Check out this code : jonas-haxe and specificly the scraper part of it. Written for Neko, with primarily undocumented classes like neko.vm.Mutex Once you have the html page you can start getting the data from it!

You will need a html/xml parser; I found one written by Yaroslav SivakovHtmlParser haxe library It also can be found on haxelib: http://lib.haxe.org/p/HtmlParser/

I found a little (old) project haxe/php project that I will post as a reference https://github.com/andor44/old_scraper. But then it stops…

Not a field that a lot of haxe-developers walk. Fun!

Update #2

  1. The htmlparser doesn’t work with the html code I am scraping. So I need to focus the parts I want to use. Regular expressions are the way to go, and I suck at them. Luckily I found a online tool that helps with testing the regex: http://www.regexr.com/ from an old flash hero gskinner.
  2. Another thing I ran into, was the data from https sites. You need something “extra” to download html files from there: install hxssl via haxelib haxelib install hxssl and add it to your build.hxml -lib hxssl

Update #1

I am coding this with openfl/regular expressions, but perhaps a better way to-go is node.js! And you can use node.js with Haxe (perhaps not completely ready: hxnodejs but probably good enough for the examples below).

I can’t really say how to start with node.js and Haxe because I have never tried it, but what I have red about it shouldn’t be a big problem. Fun again!

Read this

Some interesting reads… somewhat related to haxe

Openfl papertoy art project

I love to create projects where you take stuff from the digital world (temporarily, intangible) and drag it into the “real” world. This project is a good example of that. I create a papertoy generated in code and cut by a machine.

Besides filling my blog with new content, I have two other reasons to write this post:

  1. You can use Haxe/Openfl for something else then game-developement! I know I am not the only one, but this group of developers are not as present as the game-defs.
  2. It’s a long and complicated process to get to the end result: it’s difficult to explain this in detail to others, so I wrote down the whole story for interested friend/family/colleagues/fans???

If I ever grow a pair, this post be one of the two talks I would give during wwx2015 just to balance the all tech talks during the event.
But nothing is growing besides my hair, so instead I will write about the process and end-result.

Feedback is always nice, so please don’t hesitate to comment!

Continue reading

navigateToURL for Openfl

I have a Flash background.
And I notice that I first think (or Google) what would the code be in Flash and then try to convert it to Haxe/Openfl

Same with:
<a href="open-link-in-new-window" target="_blank">

AS3

var myURL:URLRequest = new URLRequest("open-link-in-new-window");
navigateToURL(myURL, "_blank");

Openfl

var myURL:URLRequest = new URLRequest("open-link-in-new-window");
openfl.Lib.getURL(myURL, "_blank");

Using svg for assets in Openfl

I wanted to use svg for icons in an Openfl project.
It’s not intuitive to do this, so I wrote it down for future use.

What are svg files?

Scalable Vector Graphics (SVG) is an XML-based vector image format for two-dimensional graphics with support for interactivity and animation.

Source: http://en.wikipedia.org/wiki/Scalable_Vector_Graphics

Good to remember that they scale awesomely without lost of resolution!

How to add it to your project

You can find the svg code on github.
But you can install it easier with haxelib

Open your terminal and write:

haxelib install svg

To add it to an OpenFL project, add this to your project file:

<haxelib name="svg" />

Code

import openfl.Assets;
import openfl.display.Shape;
import openfl.display.Sprite;
import format.SVG;
 
class SVGExample extends Sprite
{

  public function new()
  {
    var svg : SVG = new SVG(Assets.getText("assets/openfl.svg"));
    var shape : Shape  = new Shape();
    svg.render(shape.graphics);
    addChild(shape);
  }
}

NICE!!!
But now the cool part: it scales without losing resolution

Check the highlighted line for the changes,

import openfl.Assets;
import openfl.display.Shape;
import openfl.display.Sprite;
import format.SVG;
 
class SVGExample extends Sprite
{

  public function new()
  {
    var svg : SVG = new SVG(Assets.getText("assets/openfl.svg"));
    var shape : Shape  = new Shape();
    svg.render(shape.graphics,0,0,1000,1000);
    addChild(shape);
  }
}

for the curious: https://github.com/openfl/svg/blob/master/format/SVG.hx

Sources

  • I remembered this gist, which explains almost everything.
  • Get icons here, they are free for download as long as you credit freepik
  • Featured image from webdesignerdepot

Delay call, Timer delay in Haxe

When I started programming for as3, I started to collect little snippets of code. Some to explain the transition from as2 to as3. And some just to have one place to go to when I forgot how it worked… I decided to do it also for Haxe and Openfl. For some reason I forget some stuff easily. This is one I need often and forget easily:

How do you call a function with delay? (without a tweening engine!)

Not that difficult:

http://api.haxe.org/haxe/Timer.html

haxe.Timer.delay(someFunction, 1000);
But what if you want to send some extra parameters?

How do you call a function (with parameters) with delay?

(without a tweening engine!) I found my answer on stack.

update1 : this only works with flash, java, js, python

haxe.Timer.delay(callback(someFunction,"abc"), 10);

But I prefer a more readable version (probably because it looks very much like the javascript version), also mentioned in the same post.
haxe.Timer.delay(function () { func(arg1, arg2); }, delay);

update 2: needed it for a quick prototype in Haxe / Neko

Sys.sleep(2);
// do something delayed

Source: http://stackoverflow.com/questions/3063286/pass-arguments-to-a-delayed-function-with-haxe