Facebook’s HipHop

HipHop takes normal PHP code and transforms it into C++, which is then compiled in machine code. Code compiled by HipHop runs significantly faster than normal PHP run with the Zend Engine.

It has its disadvantages though. HipHop is only capable of transforming a subset of the PHP language. Some important (and some may say evil) features, such as eval, have been removed to make it possible to transform from PHP to C++. It also has a few other limitations, such as only supporting PHP 5.2 and 64-bit operating systems.

IBM’s WebSphere sMash

WebSphere sMash is a Java implementation of a PHP runtime environment. The PHP code is compiled into Java bytecode which runs on the Java Virtual Machine. The major benefit of running things in Java is that it is portable to more operating systems and allows you to use things like database connection pooling.

While it does support more PHP functionality than HipHop, it is still missing some features. However, it is capable of running a few major open source packages out of the box, including SugarCRM, Mediawiki, and WordPress.

Caucho’s Quercus

Quercus is another Java implementation of a PHP runtime environment.It too allows you to use Java features like connection pooling, and supports major open source packages like MediaWiki and WordPress.

The paid version allows you to pre-compile your PHP scripts, which allows them to run super-fast. Unfortunately, this is cost-prohibitive for users that have more than 1 CPU in their server (like me) or plan on having more than 1 CPU in the future (most anyone with a vaguely popular website).

Phalanger

Phalanger is a strange creature. It is sort of like PHP.NET. It takes your normal PHP code and compiles it into MSIL, which can then be run by .NET or Mono. The major advantage to using Phalanger is that it makes it possible to access .NET classes in PHP (for example, VB.NET or C#).

Thanks to Mono, .NET isn’t limited to Windows. This means you can write Gtk applications in PHP. I hope to have something released in the near future that uses Phalanger.

This short Perl script (less than 350 lines of actual code) is capable of turning your lowly desktop computer into a server killing monster. Traditional denial of service attacks use several clients (hundreds, sometimes thousands) to overwhelm the target server. The clients make as many requests as they can from the target server, causing it to use all its resources responding to the requests. Slowloris has a different approach.

Instead of using hundreds of clients, a Slowloris attack can often be successful run from a single client. And instead of overloading the server and utilizing a pile of bandwidth, Slowloris leaves the load on the target server at near zero and uses almost no bandwidth. It still makes a pile of requests, but it makes .…. them .…. nice .…. and .…. slow. Most web servers are only capable of handling a certain number of requests at a time (say, 150) so if you start 150 requests to the target server and leave those requests open for 10, 20, or even 30 minutes, you’ve effectively made it impossible for anyone else to get a valid request through to the web server. The target server isn’t actually doing any work, so the load doesn’t go up. What makes it even more annoying is the requests don’t show up in the log until they fail, if they ever do, which makes it more difficult to figure out what the heck is going on.

There are ways to mitigate the effectiveness of this type of attack, the most common being the use of a proxy to sit between the client and the web server (like haproxy or CloudFlare) and/or installing a bunch of Apache modules. Fortunately I’m doing both so hopefully I’ll be fine if someone decides they don’t like me much and wants to take down my server.

As a final note, I wholeheartedly discourage you from trying out the software unless you only do it on your local network. Attacking someone’s server is probably illegal and is likely to get your internet service shutoff. In other words, don’t be stupid.

So what is a zone file, you ask? Basically this file keeps track of how to access every .com domain name in the entire world. Well, technically not all of them, just the ones that have name servers associated with them, but practically all of them. So how many .com’s are there in the zone file? Over 88 million! Holy cow that is a lot of domains!

So the first problem is how to use the zone file. It is a 6.5 GB text file so you can’t just open it up and say “hey is jacoballred.com taken?” and expect a quick reply. On top of that, it isn’t even designed to give you a list of taken domain names, that is just a happy side effect of keeping track of how to access all the .com’s in the world.

My solution was to preprocess the data using a few Linux utilities, then load it into MySQL.

The zone file looks a little like this:

 NS E.GTLD-SERVERS.NET.
 NS M.GTLD-SERVERS.NET.
$TTL 172800
ENERCONTECHNOLOGIES NS NS1.BIZ.RR
ENERCONTECHNOLOGIES NS NS2.BIZ.RR
SELF-DRIVE-CAR-RENTAL NS NS3.IZP

None of the domains in the file have .com on the end. Each of these lists a nameserver after it. There are also non-TLD domains (nameservers) that I don’t care about, and other random markers in the file ($TTL). All I want are the domain names, so I use Linux to strip out the stuff I don’t want:

sed -e '/^[^A-Z0-9]/d' -e '/^$/d' -e 's/ .*$//' -e /[^A-Z0-9\-]/d com.zone \
| sort -u \
| awk -F "" '{close(f);f=$1}{print > "com.zone.split."f}'

Wow doesn’t that look fun? So lets go over it.

That first line uses sed to load in the zone file (com.zone) and remove all lines that don’t start with A-Z or 0–9 (the only valid characters for the first character of a .com domain), then it removes blank lines, then it removes all but the first word on each line (gets rid of the nameserver after the domain name), and finally removes any line that has characters that aren’t allowed in a domain (anything other than A-Z, 0–9, or a dash). This gets a list of JUST the domain names (without the .com), but has duplicates and they aren’t in any particular order.

The next line sorts the list of domains and removes duplicates.

The last line uses awk to split the list of domains into 36 separate files, one for each starting character (A-Z, 0–9). This isn’t technically needed but makes things more convenient.

My server is pretty wussy (1GB of RAM) so I’m preprocessing on my fast 8GB of RAM desktop at home. So I kick off that command and 20 minutes later I have files ready to be loaded into MySQL.

My table structure is pretty basic. I have 1 table for each letter/number (for performance) that has a numeric primary key and a varchar for the domain name. So I run this for each letter and number:

DROP TABLE IF EXISTS `com_A`;

CREATE TABLE `com_zone`.`com_A` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY ,
    `name` VARCHAR( 255 ) NOT NULL
) ENGINE = MYISAM CHARACTER SET utf8 COLLATE utf8_general_ci;

Next I use LOAD DATA INFILE to quickly pull in the data:

LOAD DATA INFILE '/path/to/file/com.zone.split.A' INTO TABLE `com_A` (name);

This step took about 5 minutes total for all the processed files. It is super super fast, but we still have one step left. Without an index on the name field, queries are really slow (about 2 seconds for a single domain). So we add an index to each table:

ALTER TABLE `com_A` ADD UNIQUE `name` ( `name` ( 255 ) );

This step was painfully slow, about 40 minutes, but once it was done I could do pretty much any query in a fraction of a second.

The final step was to turn off MySQL on my desktop, copy the MyISAM files to my server, then restart MySQL on my server so it could use them. Woot! I know have nearly every .com in the world on my server, ready to tell any web app I want if a domain is available or not with a high degree of confidence. I have a couple really fun webpages in the works that will use this.

Well that was a bit of a ramble but should be enough to get someone else in my position on the road to domain goodness!

So, not really understanding exactly what it was, I signed up and configured it for the Fake Name Generator.

After some poking around and letting it do its thing for a few days, this is what I’ve discovered it does for me:

• Provides a free DNS management. This is included for free with many registrars, but it just so happened that the domain I’m trying this out on didn’t come with DNS management so I’ve been paying $10 a year for it. This alone makes CloudFlare worth using for me.
• Serves my content on a CDN-like intrastructure. This makes my site faster to some users, which is always a good thing.
• Caches my static content (like images and JavaScript). This dramatically reduces my server load, and makes my site faster. My LAMP server with only 1GB of RAM is currently serving about 100,000 pageviews per day and running millions of queries in offline processes. With CloudFlare, my load average rarely goes above 0.10.
• Blocks bad guys. This is a huge deal for me. Everyone and their mom thinks it is okay to scrape my site for data. Bots love to hit my site to try to find exploits. CloudFlare does a great job at identifying these people and blocking them for me, or providing a way for them to enter a captcha to prove they aren’t a bot.
• Provides geolocation data on all visitors. I haven’t started using this yet, but CloudFlare adds a request header with the visitor’s geographic location. This makes it easier to target content to visitors from certain parts of the world.
• Makes me more profitable. All around, CloudFlare has made my business more profitable. My site requires less server resources, which means I can keep my site running on my relatively cheap tiny server. Fewer bots are loading my ads, which means my click thru rates are higher, which means I get paid more. My pages respond faster, which means I’m ranked higher in the search engines, which means I get more visitors.

One problem I ran into, however, is occasionally a screen scraper gets through their blocks and starts hitting my site. In the past I would use iptables to block them, but the way CloudFlare works makes that impossible (at least with my limited knowledge of iptables). CloudFlare provides a way to block a specific IP, but it can take several minutes to go into effect.

The solution I came up with is to use Apache to give visitor’s from the offending IP a 403 error:

<VirtualHost *>

 SetEnvIf CF-Connecting-IP 98.17.241.185 GoAway=1

 <Directory "/path/to/your/website">
 Order allow,deny
 Allow from all
 Deny from env=GoAway
 </Directory>

</VirtualHost>

This snippet, properly placed in the Apache config file, will cause Apache to look at a header set by CloudFlare, and if it matches the offending IP (in this case 98.17.241.185), it denies access to the site. You can add a nearly unlimited number of SetEnvIf statements to block any number of IPs.

Anyways, if you get an invite to CloudFlare, check it out! It is definitely worth it!

I’ve been wanting to make a Firefox add-on FOREVER, but have never found something worth making that hasn’t already been made (or isn’t way beyond my abilities to make). So I decided I’d start simple and make a search provider add-on for ABA Number Lookup.

I found a page in the Mozilla developer wiki that explains how to make an OpenSearch plugin. It is super easy. Basically it is just a snippet of XML that tells the browser where to send queries to, and how to get search suggestions. It also contains basic information like the name of the search engine and who wrote it. Easy stuff.

It took about 30 minutes to modify ABA Number Lookup to be able to return search suggestions and to send the proper MIME header for the OpenSearch XML file. After that was done, I went to the “submit an addon” page, followed the instructions, and wham! My search engine add-on is available to the world!

Mozilla puts all new add-ons in a sandbox to make it a little harder to push malicious code. This is just a search add-on so I think it should pretty quickly get approved and the download page will look a little less scary.

Anyways, it was fun! I might write one for Rhyta Arcade just for fun, though I doubt anyone would actually use that one.

Google has this awesome alert service that lets you set up search queries and have new search results from them emailed to you automatically. For example, I have “jacob allred” as a query, so anything new with my name in it pops up in my email.

On Saturday night, I got an alert email from Google for “fake name generator”. Turns out somebody made a Fake Name Generator Android application and put it in the Android Market. “Well that is strange,” I thought to myself. “I don’t remember giving anyone permission to steal my content.” The guy stole my product name, logo, look and feel of my site, layout of my site, and my content. And to make it worse, his app has an ad on it so he is making money off my product while I’m stuck paying the bill for the server that powers it.

I decided to try to be a good person first, and try to contact the developer and see if we can work something out. Turns out the Market doesn’t really require developers to provide contact information. Lame. I managed to find an email address but I’m not sure if the developer actually reads it. I sent an email to him letting him know I’m not thrilled about his app and that I’d prefer to work out a way for him to continue using my services (using the API instead of screen scraping) in a way where we can both get paid.

Next step is to try to notify Google of the copyright infringement. Google has detailed instructions on how to file a Digital Millennium Copyright Act infringement notification, but it requires either faxing (I don’t have a fax) or mailing (which takes days). Neither is a great option.

My last option is to try to break the app. This requires downloading the app and then viewing my logs to see what it looks like when it scrapes my site. Should be easy, right? Not exactly. I don’t have an Android phone and apparently the Market isn’t available on the emulator. Lame.

Luckily, with much searching, I found this blog post that explains how to get the Market working in the emulator (albeit with an older API version). I ended up having to download the 1.6 image he posted on 4shared.com, but at least it works. Once I got the Market working, I quickly downloaded the app and broke it. That should get the developer’s attention. ;)

Anyways, if you are an Android, iPhone, or Palm app developer and want to make a legitimate fully-sanctioned application, shoot me an email. I’m interested in paying someone to write apps for all of those platforms, or in working out a revenue sharing option.

The first step was to find some data. I was planning on using Ignyte Software’s free movie API, but that recently stopped working. Bummer.

I decided maybe I’d use isnoop.net’s movie RSS feed, but they got shut down by Google.

After reading online, I discovered that the big companies that collect theater data can’t copyright their data (because it is a collection of facts) and so it is legal to screen scrape and display the data elsewhere, but I hate screen scraping other sites (because I hate it when people screen scrape me) and it doesn’t matter whether it is legal or not, Google can still sue me into oblivion for doing it.

So my last option was to purchase the data. This took a lot of work to figure out who even sells the data. After some digging, I found that there are 2 companies in the US that sell theater showtime listings: West World Media and Tribune Company.

Both offer a data feed that you can download that includes all the showtime listings for the US. Both want $1500/month for this privilege. That is the minimum, the pricing scales based on usage.

Both offer an API you can call to get listings just for a specific zip code. While this is cheaper (around $500/month if I remember correctly), you also have to pay for each query and you can only search by zip code.

So the end result? No theater showtime listings website. Wayyy too much money. If you happen to be rich and can drop $1500/month for such a thing, I’d recommend West World Media. You get way more for your money (including movie reviews), and they just seemed to have their act together better than Tribune.

Here is how it works:

1. I loaded several popular Scrabble dictionaries into a table.
2. I loaded Wiktionary into a table.
3. I created a form that lets a user entered a scrambled word (like EMURLNRASBC).
4. When the user clicks “Unscramble”, I unscramble the word for them:
   1. I query the dictionary table to find all words that contain all the letters (and the proper amount of each letter) and is the proper length. So in this example, we need all words with two R’s and one of each of those other letters, and it needs to be eleven characters long.
   2. I join this list with the Wiktionary table and find out how many definitions there are for each word.
   3. I sort the list so the words with the most definitions are on top. This helps a little to prevent obscure words from being listed first in situations where there are several words made up of the same letters.
5. I display all the words that are made from the scrambled letters.

It didn’t take too long to make, and my wife made an awesome logo for it.

So what does EMURLNRASBC unscramble to? Visit Word Unscrambler and find out!

The most recent thing I am trying is image optimization. With the announcement by Google that they are adding site speed to their page ranking equation, it is more important than ever to have a fast site.

I get all my games and game thumbnails from Mochi, a free source for Flash games. I’ve written code to do this all automatically for me. The problem is, some game authors think it is okay to include a 200KB thumbnail (they are only 100x100px!). Alone this might not be the end of the world, but when you get a page of 50 thumbnails and each is 60KB to 200KB, then you have a problem.

The solution? Image optimization!

It is important to note that image optimization is not image compression. Compression generally results in degraded image quality. Optimization programs take the image file that already exists, and re-organizes the data in the image file so that the end result is the same, but it takes up less space. Optimization programs sometimes also remove data that isn’t relevant to the display of the image, like unused colors in a GIF color palette or EXIF data in a JPG.

I had a really hard time finding a JPG optimization program that didn’t reduce the quality of the picture, but finally found a 5 year old package that does the job great: littleutils. Note: The download link on that page is wrong, try this one instead.

This collection of little utilities includes 3 great image optimizers for Linux: opt-png, opt-jpg, opt-gif. These items do a brute-force style optimization of the image, trying many methods until it finds the method that produces the smallest file size for each individual image. Even though these tools are 5 years old, the opt-png utility produced smaller file sizes than the popular pngcrush and pngout utilities, and the punypng web service. Amazing!

By using these tools, I reduced the file size of my thumbnails by an average of 35%! That is a huge difference, and will make a huge difference to my visitors.

I’ve also updated my automated game downloading script to tie into these utilities, so the optimization happens without me having to ever touch it.

If you need help installing these utilities, let me know. There were a lot of optional dependencies that made a huge difference in the size of the optimized files, so make sure you install all of them before compiling littleutils.

What I really want to do with it is to dynamically generate avatars to be used on the Fake Name Generator. For example, the code would pick a skin color from a predetermined range of skin tones, a hair style, a hair color, lip style, lip color, eye shape, eye color, etc etc. It would then layer all these items together into an SVG graphic. Using a bit of JavaScript, I could even make the eyes blink every few seconds. If needed/desired, this SVG graphic could then be converted to a PNG image.

But that is a bigger task than I’m ready to really start at the moment, so instead I’m playing around making a simple matching game. The goal is to create a mix of PHP and Raphaël to allow the developer to define a list of images that are used to automatically create a matching game. Unlike most matching games available online, mine will allow the developer to specify two different images that match. For example, a picture of a person and an image with the person’s name would be a matching set. Or a picture of state and a picture of the state’s flag. Things like that.

I haven’t gotten very far yet. As of this writing, all it can do is generate 3 pairs of cards. Each pair of cards has a different color. The game lets you click cards and try to match them. Very basic.

Once it is finished, I plan on releasing the source code under a friendly license and then using it on an LDS gaming site I’m working on.

Anyways, check out the matching game or the demos at Raphaël.