Some days into the project Lambda added Node 4.3 support. Previously the only supported Node version was 0.10 and the lack of generators made the code super messy. Without Node 4.3 this blogpost probably would look kinda different.

Lambda is not a web server!

The signature of a Lambda function:

1
2

exports.handler = function(event, context, callback) {
}

Lambda functions can be triggered from different sources (S3, DynamoDB, Scheduled Events and more). For processing HTTP requests another service is needed: API Gateway.

API Gateway is not tied to Lambda and can be used for various kinds of backends. Bottom line: The API Gateway/Lambda stack has very little to do with something like Express. On the contrary, mastering API Gateway is hard and I underestimated that. As a matter of fact, most debugging and cursing happened on API Gateway side and not because of Lambda.

How to get requests into Lambda

API Gateway provides a graphical interface where you (basically) setup an endpoint and method and map it to a Lambda function.

By default API Gateway only passes the request payload (in case of a POST/PUT) to the event parameter (see above) of a Lambda function. And for a GET request nothing is passed!

Input mapping

In order to get all the request meta data into Lambda (like path, method, ip, headers etc.) an input “Body Mapping Templates” must be set up. This is the mapping I used:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

{
  "method": "$context.httpMethod",
  "body" : $input.json('$'),
  "path": "$context.resourcePath",
  "ip": "$context.identity.sourceIp",
  "headers": {
    #foreach($param in $input.params().header.keySet())
      "$param.toLowerCase()": "$util.escapeJavaScript($input.params().header.get($param))" #if($foreach.hasNext),#end
    #end
  },
  "queryParams": {
    #foreach($param in $input.params().querystring.keySet())
      "$param": "$util.escapeJavaScript($input.params().querystring.get($param))" #if($foreach.hasNext),#end
    #end
  },
  "pathParams": {
    #foreach($param in $input.params().path.keySet())
      "$param": "$util.escapeJavaScript($input.params().path.get($param))" #if($foreach.hasNext),#end
    #end
  }
}

These values are then available in the event parameter of the Lambda handler function.

How to get responses out of Lambda

For a success response it’s quite easy. From Lambda do:

1
2
3
4
5
6

exports.handler = function(event, context, callback) {
  // do the work here

  let responseData = {};
  callback(null, responseData);
}

response is what your user will receive as the response payload.

How to setup non-default (200) responses

Probably the most annoying thing with API Gateway (and the thing that produces the most Lambda related issue in the API Gateway forums) is how to setup custom HTTP response codes. It works by regex’ing over the error reponse from Lambda.

Suppose you want to respond with a 404. From Lambda you call:

1
2
3
4
5

exports.handler = function(event, context, callback) {
  // do the work here

  callback("404 happened");
}

Since the first parameter of the callback function is used, API Gateway understands it as an error. In API Gateway you setup a “Lambda Error Regex” with regex .*404.* and the user will receive status code 404.

So how to not only set the correct status code but also pass a response to the client? You can pass the status code and response body as separate properties:

1
2
3
4
5
6
7
8
9
10

exports.handler = function(event, context, callback) {
  // do the work here

  callback({
    status: 403,
    body: JSON.stringify({
      message: "you're not allowed to do that"
    })
  });
}

In API Gateway I then setup a “Lambda Error Regex” where the regex is .*\"status\":403.* and the following output “Body Mapping Template” (“Body Mapping Template” maps the reponse from Lambda to the response body that is passed to the user from API Gateway):

1
2

#set ($errorMessageObj = $util.parseJson($input.path('$.errorMessage')))
$errorMessageObj.body

$.errorMessage is the error data coming from Lambda.

This needs be done for all response codes. You’ll end up with something like this:

How to set response headers

Suppose you want to set a set-cookie or a location header. How to do that? You can set “Header Mappings” where you map a property from the Lambda response object to a header set in API Gateway.

Here is the catch: that does only work with the default response (see 200 from above) but not with the “Lambda Error Regex” workaround since the “Header Mapping” can not access the $.errorMessage property.

That means that you can currently only have one response code (the one that you defined as default) to set headers. I’m sure AWS is currently working on that. Because this is bad.

Get used to swagger config file early

While the graphical user interface of API Gateway is fine at the beginning when starting to work with and understanding API Gateway, it gets super annoying down the road because you are ending up copy-pasting stuff all the time. For instance you will likely want to have the same “Lambda Error Regex” for most of your endpoints. And you definitely need to have the input “Body Mapping Templates” from above on every endpoint.

I resisted way too long against using a swagger config file, since I was unfamiliar with it and thought I already need to learn enough new stuff. However, it’s easy to understand since it provides the same settings as the graphical user interface. I suggest you set up your basic settings with the GUI and then export it via “Stages” –> “Your stage name” –> “Export” –> “Export as Swagger + API Gateway Extensions”.

Then you can edit the swagger config file locally and push it with:

1

aws apigateway put-rest-api --rest-api-id <your_api_id> --mode overwrite --body file://<path_to_swagger_file.json>

Serving static files

Most parts of the site were dynamic (the admin interface and the API), but there were also static ones (the homepage). API Gateway can be used to serve static content by using the “AWS Service Proxy” Integration type. With that static files can be proxied from S3.

But it’s slow! And requests count against the API Gateway throttling limits. Instead you should use subdomains for requests that should be handled by API Gateway Lambda (i.e. api.your_domain.com and admin.your_domain.com) and map the homepage (your_domain.com) to CloudFront. This is specially true since API Gateway can not server binary files (i.e. images).

Note: I did not test the API Gateway cache.

How to run Lambda functions locally?

I didn’t do it. Being able to run the tests locally was enough for me. Final tests I did on production. I understand that this would not be suitable for larger and more complex projects.

What to use for deployment?

I went with node-lambda. I haven’t really tested the alternatives. node-lambda does what I needed (deploy the function). Their ENV variables handling is nice.

At scale

If you’re expecting a lot of traffic you might want to contact AWS support to increase API Gateway and Lambda limits.

Lambda is smart in reusing the spawned Lambda nodes. So if your Lambda function executes relatively fast (~300ms, typical for API requests) and the traffic is moderate the default Lambda limit (100 concurent Lambdas) should be fine.

No gzip

:(

Conclusion

While Lambda is easy to get used to, having to proxy requests through API Gateway makes everything hard that is usually super-easy in plain node (with the help of express or friends). Now that I went through all the pain, everything seems obvious. But it was a hard process.

Still, serverless is the new cool kid on the block and I enjoyed not needing to deal with provisioning servers (and make them scale). AWS is the leader and they are actively improving Lambda and API Gateway.

I’m also very excited about zeit.co. With them you can just deploy your express app. Need to look into that soon.

• What Is Functional Programming?
  Must read.

• Which Programming Languages Are Functional?
  Must read (continued).

• the changelog #187
  With Dan Abramov, creator of redux.

• Edge conference videos are online
  “Relive every second of Edge with complete video coverage”. Will do.
• Code Guide by @mdo
  We are also planning to create Code Conventions apdev edition
• What is a Heisenbug
  Conquer the Heisenbug, one of the most satisfying things in programming.
• How much damage does third party content really do to e-commerce sites?
  Some additional info to our own post about 3rd party content
• 10 career-changing programming books
  If you still need to read some more about programming … maybe there’s a book on this list, that you haven’t read already.

I wanted to take look at Grunt for a while now. Finally, a couple of weeks ago, I ran into a problem that I couldn’t overcome with NPM and its scripts section alone. The right time to check out Grunt, I thought. Has been too long on my TODO-list.

The problem I had to handle was: deployment of a node project to my dev server. And I didn’t want to rely on git hooks or CI for this. I wanted a script I could call that uploaded everything to my dev server via ssh, ran npm install and restarted the application (more on this in a seperate post later).

After googling for “deploy grunt” and reading through the gazillion results, it became quickly clear to me, that I wasn’t the first person that wanted to try and do this with the help of the boar (or whatever this thing is). I installed about every grunt plugin I could find but none fit my needs exactly … so what now? Fork another plugin and adjust it to comply with my requirements? Write my own plugin from scratch?

No, no, no. I might want to reuse this later on with a different server (no ssh) or deploy a frontend-project (not a node one) and would have to update/extend the plugin to accomodate this. Most authors of the aforementioned plugins have done exactly that – they wrote a plugin that did what they needed 100%. But because we as developers are very biased and only want to do it “our way”, these plugins are all the same but at the same time different. And, of course, I didn’t like a single one of them :).

Ok, so no plugin. What then. I decided to write a simple Grunt task that used other node modules (not Grunt plugins) to accomplish everything I needed. In this, I could reuse code that others had written – maybe not for this use-case but as a standalone module that was so granular that it “only” solved one problem but that perfectly.

Don’t write plugins that try to solve too many problems at once.

Write small modules that run in all kind of environments and solve exactly one problem. This way, others can use your code – even in scenarios which you haven’t been thinking about while writing the code … this is a good thing.

Funny enough, a couple of days after my first stint with Grunt, I listened to the JavaScript Jabber podcast and they were talking about Gulp (and Grunt, of course) with Eric Schoffstall – one of the developers behind “the streaming build system”. It was really interesting and you should definitely go and listen to it (after you’ve finished reading the post).

I don’t want to talk about performance or other important reasons why A is better than B, but there are two things – in my eyes – Gulp (or the Gulp community) does better than Grunt:

1) There is a curated plugin list

So, we’re curating it. We don’t just come out and say, “Oh this plugin sucks. We think it’s stupid. You’re not allowed on our website.” But we do say, “This plugin doesn’t work,” or, “It has too many bugs. It just shouldn’t be a plugin. And those are our requirements.”

So there shouldn’t be duplicate plugins or plugins that wrap some functionality you could use directly via its node module

2) Recipes instead of too many plugins

[…] we actually have a section of docs called recipes. And these are just, “Hey, I used Mocha with Gulp. Here is a task that I used it in and let me explain why I did a couple of things.” So that way, people can just say, “Okay, I want to use Mocha. Let me go look at the recipes and go find one for Mocha.” And it’ll explain in-depth why they did what they did, how to use it […]

If there is a problem you solved by using already available node modules, share it with other devs that might get stuck at the same point. But don’t do it writing your own plugin – write a small HOWTO instead.

Don’t write code that can’t be used outside of Grunt/Gulp

Cheers.

• Cookieless Domain vs. DNS lookups
  Finally some stats regarding that issue.
• Gulp vs. Grunt
  The guys from the JavaScript Jabber podcast talk to Eric Schoffstall, one of the creators behind gulp.js. Really interesting background info why they created a new build-system (instead of just using grunt).
• Stephen Wolfram’s Introduction to the Wolfram Language
  Well, we should have a look at it, right?
• ATOM
  New “hackable” text editor from the people we love @ Github. Can’t wait to get our hands on this one.
• Salted Password Hashing – Doing it Right
  Some more input on securely saving passwords – as an addition to the article from last week: salt’n’pepper – or how to store passwords securely

Fun fact: You can check who added what link by checking the commits.

Accessed information included usernames, email addresses, mailing addresses, phone numbers, and encrypted passwords. Actual passwords were not revealed, […]

Good to know, that they didn’t store the password in plain text but encrypted the most sensitive data.

You see, in today’s world it is very important to make sure that an attacker, after aquiring your user data, isn’t able to just know you password and:

1) log into your account
2) more importantly – use the password for other services with which he/she could really “harm” you

[…] however it is possible for a malicious person with enough computing power to guess and crack an encrypted password, particularly a weak or obvious one.

That’s true – even when using encryption. So, we as developers have to make sure that this process of “computing” or “guessing” takes as long as possible (and has to be repeated for every password).

I can remember my first attemps at server-side user-management with PHP 13-14 years ago. I just stored the user’s data like this:

----------------------
| user | password    |
----------------------
| phil | topsecret   |
| aron | notsosecret |
----------------------

The user sent the data via POST to the server where I would compare the password he/she just entered with the password stored inside my database like this:

// don't do this
if ($_POST["password"] === $row["password"]) {
  // grant access ...
}

If the user ever forgot his/her password, I could just get it from the DB and resend it via email :( – a really bad idea! But back in the days I didn’t know better.

Rule #1
Never save passwords in plain text

Use a hash function to store the password securely. Hash functions are one-way algorithms that turn your plain text password into a totally unrecognizable number and letter combination. Plus they produce results that differ completely even if the input is almost the same.

The following snippets are JavaScript because our last project was a node.js project – but these functions are available in other programming languages as well.

You get the idea …

But this is not enough. If we save the once hashed password into the DB, hackers can use methods like “dictionary”–, “brute force”-attacks, lookup– or rainbow-tables to “guess” your password. It’ll take some time, but they will get there – and even faster if the original password only contains letters from a-z.

Rule #2
Salt your password before saving it

Hashing function always produce the same result if the input is the same. You can run the code above as many times as you like, the result won’t change.
Two users with the same password would get the same hash, which makes it easier for an attacker because he/she can use lookup- or rainbow-tables to “decrypt” your password.

Appending a random string (a salt) to the password makes this impossible.

I don’t know if they still do, but in a previous version wordpress used the hashed SITE_URL as a salt for every password – not good. You can do better than that.

Create a new salt everytime you store a password in the database – never reuse an old one. And don’t make the salt to short. It gets harder to hack the longer the salt is.

var crypto = require("crypto");
var salt = crypto.randomBytes(64);

If you don’t use the same salt for every password, you have to store the salt in the database as well. Either in a seperate column or add it to the password as “metadata”.

------------------------------------------
| user | password                        |
------------------------------------------
| phil | <hashed password + salt>:<salt> |
| aron | <hashed password + salt>:<salt> |
------------------------------------------

Salting the passwords prevents hackers from using lookup-tables to gain access to your passwords quickly but with up-to-date hardware they can use “brute force” to generate billions of hashes a second and compare them with the hashes you stored in the database.

Rule #3
Use “slow” hash functions

This may sound strange but another method to make sure it’ll take attackers more time to hack the passwords is to use an encryption algorithm that takes it time to produce a result. This reduces the number of hashes a hacker can generate to a fraction of the number above.

PBKDF2 is a good choice for this.

Storing the encrypted password

Retrieving the encrypted password

Hope this summary helps and makes your next project a bit more secure (than it already is).

• Responsive Web Design Performance
  Etsy’s Lara Swanson talks “mobile first” (Design + Performance)
• Performance for Responsive Web Design
  Free webcast presented by Maximiliano Firtman
• JavaScript Array: slice vs splice
  Convention of how to distinguish between modifying and non-modifying object method calls.
• Maker’s Schedule, Manager’s Schedule
  Paul Graham’s piece about “flow” is already 5 years old but still true today.
• Inventing on Principle
  Great talk by Bret Victor about how our tools (or programming in general) should work nowadays.

YMNNJ is a collection of vanilla JavaScript snippets that can (more or less) replace some often used jQuery built-in functions. Zack Bloom and Adam Schwartz, the guys behind YMNNJ, want us to think twice before using a (big) framework/library like jQuery:

If you’re developing a library on the other hand, please take a moment to consider if you actually need jQuery as a dependency. Maybe you can include a few lines of utility code, and forgo the requirement.

I totally agree with this statement. You should pause and ponder before using something like jQuery (or e.g. Modernizr), be it in a library or any other project you are doing. It comes with a cost that you definitely should be aware of.

Tim Kadlec also wrote a post and sums up the performance hits pretty well:

This is concerning, but it’s not just download sizes that you should be worried about. In a presentation given at Velocity in 2011, Maximiliano Firtman pointed out that on some phones (older, but still popular, BlackBerry devices for example) can take up to 8 seconds just to parse jQuery. More recent research from Stoyan Stefanov revealed that even on iOS 5.1, it was taking as many as 200-300ms to parse jQuery.

But jQuery isn’t all bad, of course not. Fittingly, Webplatform Daily posted a link to a list by Rick Waldron of all the quirks jQuery fixes for us developers that need to address as much devices/browsers as possible. Try fixing (or even finding them in the first place) them in vanilla JS by yourself.

So, if you’re aware of the “downsides” of something like jQuery but decide to use it anyway (because of all the “upsides” it includes), it’s totally fine. But think of ways to use it responsibly.

Again taken from Mr. Kadlec’s post:

[…] some of the devices we needed to test on couldn’t load the page at all if jQuery was present — it was just too much JavaScript for the device to handle.

In one of our last mobile projects we decided to rely on jQuery for all the more fancy things we wanted to implement. Features that the older phones couldn’t or shouldn’t cope with because they would crash or suffer devastating performance hits that would keep the users from navigating the site. Using Progressive enhancement and a technique like the BBC’s “Cutting the mustard” allowed us to only deliver JavaScript (including jQuery) to the more capable devices – avoiding to make your beloved Blackberry explode :)
In addition, the user only had to download HTML, CSS and assets to view the page, which saved about 40-50KB of transfered data and some JS parsing/execution time.

Tim’s closing paragraph:

I’m not saying that we stop using libraries altogether—and neither were the people who created “You Might Not Need jQuery”. I’m suggesting we make that decision with a great deal of care.

WORD!

• DevSwag
  Get stickers from open source projects. Support and promote them.
• YOU MIGHT NOT NEED JQUERY
  I love jQuery when I need it, but I don’t always need it.
• Sass: Mixin or Placeholder?
  Understanding the difference.
• Map caps locks to control
  Do it! ++productivity
• Do Things that Don’t Scale
  Great stories about famous startups and how they got to where they’re now.
• Are coders worth it?
  “In today’s world, web developers have it all: money, perks, freedom, respect. But is there value in what we do?”

So we developers take care of:

• gzipping
• minify/uglify + concat HTML, JS and CSS
• all blocking JS at the bottom of the page
• caching/CDN
• fewer requests
• and a lot more …

Each of the points above will improve your site’s performance – sometimes more sometimes less, depending – of course – on your site and its structure/content.

In one of our last projects we took this to heart and wanted to deliver a beautiful and fast mobile website (m-dot-domain, so it was really only mobile). We’re not talking about the “under 1 second to screen” scenario, but rather a smooth surfing experience using a 3G network. Let’s say something like 1-2 seconds until “start to render” and ~10 seconds for “loaded”.

Starting with HTML templates/components that – if needed – get progressively enhanced was a good foundation for this. With dummy content filling the HTML, this is a breakdown of the filesizes (gzipped) by type:

• HTML ~15KB
• CSS ~15KB
• JS ~50KB (biggest part being jQuery)
• images ~500KB
• fonts ~45KB

Putting JS at the bottom of the <body> made the browser only wait for the HTML and CSS before beginning to render the page. Sadly, because we’re using webfonts, it begins rendering the content without displaying the text. Only after the webfont is downloaded, does the text become visible – in my eyes webfonts’ biggest disadvantage.

I used my slow vServer to get this waterfall, so an optimized setup with an additional CDN (like the live-server) should be even faster.

As we can see in the waterfall above – “start render” lies somewhere between 2-3 seconds and the webfonts are loaded a second later. This way the user should be able to start reading after 4 seconds. So far, so good.

This is also the moment the JS is loaded (async) and executes. Some of the components “change” a bit because of the progressive enhancement kicking in but nothing so serious that it’ll annoy the user too much.
And this is only for the user’s first visit – the second time the cache will be filled with our CSS, JS, images and fonts and the page load time should be even faster.

Because our client uses a big fancy CMS, this is the point where we bundled up our assets and handed them over to them. Their developers had to integrate our templates and scripts into the colossus that is their CMS and most of all, they also had to add things like:

1. tracking and
2. advertising
3. real content

From here on out it got messy …

Tracking shouldn’t have been a big problem, you might think. No, it usually isn’t. But they had to include more than one tracking-script and not at the bottom of the document but the head (I think we could check and move them to the bottom, though). These are mostly small scripts that load fast but every request hurts on mobile.
There are services like Segment.io but they are US-based and we Germans don’t like to send our data across the pond ;) Besides, I don’t think they support german newspapers’ most important tracking-provider IVW.
Nonetheless are these kind of services a great tool for improving your website’s performance: one call instead of two or three every time you want to track something = big win!

On to advertisement. Because our client offers all its content free of charge, ads are the only source of revenue they got. So these are important and we have to make sure they are loaded and displayed correctly.
The setup gets even more complicated because the client uses a “middleware” that is based on DoubleClick as the technical foundation – leading to more and more scripts that have to be fetched from different servers (double ouch).
DoubleClick offers an <iframe>-based “async” version of the GPT but we can’t use it because the different ad-formats won’t play well with iframes = even more synchronous JavaScript stopping the page from being rendered :(

Most of the time the first ad is located on the top of every page, so the aforementioned scripts have to be loaded before this (in the <head>). Not 1, not 2, but 4 (in words: FOUR) scripts are loaded before the custom initialisation is run and the first ad can be displayed on page … ah no, I forgot to mention that every ad loads one or more JS files to get the content they need. S-Y-N-C-H-R-O-N-O-U-S-L-Y.

All these tracking and ad scripts push the “start render” back 4.5 seconds which leads to readers looking at a blank screen for 8 seconds. If you’re lucky, they haven’t left the site when the content starts trickling in.
I’d also like to point to the fact that the webfonts are loaded long after the “start render” at around 13 seconds which means that even though the browser starts to render the page, copy isn’t displayed until they are downloaded completely.

Let’s take a look at how this changes the page’s waterfall:

Concluding points 1 and 2 we have to realize that many 3rd party content providers just don’t care about performance – at least not as much as we developers would like them to.

The last thing I want to talk about is point 3: real content.
In our case we built templates for a home-, overview- and article-page using dummy copy and images that nearly reflected real-life content (at least the amount we thought made sense considering our performance budget).
The client was in too much of a “desktop-mode” and filled the landingpage with everything that came to mind. So we ended up with around 50 article-teasers and multiple more complex gallery widgets etc.

I don’t say that it was there mistake (alone), we should have sat down with them more often and talked about how peformance is a feature and how it (sometimes) is more important than an abundance of different content elements put on the same page.

Points 1 and 2 destroyed our time to first render and 3 let the page get way too heavy for a “smooth surfing experience using a 3G network”.

Let’s not bury our heads in the sand and instead learn from a project like this. But sometimes complaining again and again is enough to make all those shitty 3rd party widgets/scripts improve. At least I hope so.

Maybe we got something you haven’t read/watched before:

• Introduction to NoSQL by Martin Fowler
  Great introductory video about NoSQL and its rivalry with other RDBMS.
• Webinar: Getting Started with MongoDB – Back to Basics
  If you want to dive into developing applications backed by MongoDB.
• Pete Hunt: React: Rethinking best practices — JSConf EU 2013
  Very tempting their approach of always be able to render the current state of the application. Their virtual DOM sounds like voodoo. We need to do a real life test.
• A game of inches
  Ever worked two days on an optimization of a feature? An optimization that most users probably won’t notice and is – in manager terms – very minor? You worked on it because it makes a difference. Because software development is a “game of inches”.
• Everpix – the story
  “why the world’s best photo startup is going out of business”
• Everpix intelligence
  Uncensored Everpix metrics, financials and business data for your perusing

Blogging with Jekyll + S3 + CloudFront
Quick Tip for Easily Deploying Octopress Blog on Amazon CloudFront

One open thing was to enable gzipped content. Zipping is certainly the most obvious performance optimization. Usually the webserver takes care of it, but we don’t have a webserver here. We have a CDN and like every CDN CloudFront doesn’t “process” request, it just serves files. Meaning that by definition it does not compress files when requested by a client. However that only means that we need to create zipped files when we deploy the Octopress files. A very good starting point:

gzip your Octopress

I made some modifications regarding CloudFront. Also rename is not available on OSX so I changed it around a bit.

But with all these advantages there also come challenges. One is positioning inside the book. So what to do, when you want to save the user’s current reading position, so that he can continue on the same “page” when he comes back? Since the user can read the text on different devices with different screen resolutions and different text settings (e.g. font size settings) you can not simply save “Page 36″.

Last read page (rough position)

If you just want the rough position (i.e. no certain text) the plain percentage is a way to go. The monocle framework (that we use at Nimbi and PaperC) does exactly that. It saves the reading position by dividing the current page number by the total number of pages (base on the rendering on the current device). On a smartphone the user might be on page 50 of 100 pages total where he is on page 6 of 12 pages total on his desktop browser. Both lead to roughly the same position.

We must be aware that there is a certain cognitive load for the user to find his actual last sentence on a page when changing devices or even changing from portait to landscape mode on his tablet. We have no chance to save the actual last words the user read.

Certain text

When it comes to saving annotations we need something more accurate. The user wants to highlight a certain sentence. We need a way to save the exact text so we can allow the user to navigate to that text directly and visually highlight the text.

Text and offset (or index)

Just saving the selected string as text and do a full text search when trying to find the string won’t do the job, since finding the actual word is impossible when the user selected a word like e.g. “and”. So together with the actual string we need at least an offset from the beginning of the text (which is usually a chapter in ebooks, also called component).

This is a typical way to deal with search results. Search engines usually deliver results based on full text search. When they index a HTML document they strip away all markup leaving the text only. So what they can give you is the found string and an offset (numbers of characters) from the beginning of the document. A result for a term like “and” (which is usually a stop word in search engines) will contain different offsets for the same term.

The bad news is, that in order for the offset to be accurate, you need to make sure that your local text-online version of the HTML document is exactly the same as the one your search engine indexed. You need to know exactly how/if the search engine converts linebreaks to spaces and how the search engine deals with spaces/linebreaks between DOM nodes (and other things). And then you have to apply these rules to the implementations on your client.

To avoid the “offset-hell” you could tell your search engine to deliver an index of the appearance in the document instead of an offset. Assuming you have three appearances of the term “and” the search engine could deliver the index 0, 1 or 2 to a result dataset. With that you are always certain which term you are dealing with.

Either way it’s not trivial to first convert the markup to text-only and then, after you found the term, find this term back in your DOM tree. Also note, that operations on huge amounts of text can be CPU-intensive, especially on mobile devices.

DOM based

Back to the case where a user wants to highlight a portion of text. Why not letting the DOM help us a little. We can make use of the Range obj.

“The Range object represents a fragment of a document that can contain nodes and parts of text nodes in a given document.”

That seems like the perfect fit for our use case. And it is! While it’s not supported by every browser environment it is well supported on engines that are typically used for displaying ebook (epub) content. For our “and” example it would look like this (jQuery pseuso code):

The range of a more complex text (that runs over more that one DOM node) could look like this (user selected “Prejudice is a book”):

This is a good starting point for describing fragments of text embedded in html. If you want to locate the visual position of this range you can call

How to save it in the backend

For saving the dataset to the backend we need a string representation of the DOM elements. Different approaches are possible. As XPath it could look something like this:

Or, if you don’t want to rely on elements to have id’s or classes:

Yap, in Xpath the index starts with 1.

However, for Nimbi and PaperC we tried to keep the data smaller (it’s potentially big data, right?) and query language agnostic:

startContainer and endContainer are the first textnode in the frist childNode of body (i.e. body.childNodes[0].childNodes[0]). The dataset is smaller (less to send over the wire and to save in the db) and – more importantly – it does not rely on XPath. While XPath might not be the bottleneck in this kind of application, it is known for not necessarily being fast, especially on mobile devices. And there is no XPath on Android < 2.4.

Here is an example of a more complex DOM tree:

This pretty much solved the position issue for us.

Pros

The position is 100% unique. You’ll find the fragment no matter the screen size or font settings. You can even save the position of a single character.

It makes use of the most basic HTML. The DOM tree was there since Day 1 of the HTML standard. So even the oldest and most outdated html parser should be able to find the fragment.

It’s query agnostic. You don’t need XPath or other stuff.

Very small data footprint.

Cons

Since the position is described in a DOM tree, the backend needs a XML/HTML parser to “understand” the position. Asume that you want the backend to create a “heatmap” of the book, where the users set highlights, where they stopped reading and so on. The backend will have to convert the position to it’s own positioning format (if it is not using the DOM tree in the first place).

Also the implementation relies on a valid DOM. Epub components (a html chapter of an epub) are typically XHTML. If they are not served with mimetype application/xhtml the browser will fall into quirks mode. When you now have DOM elements that are only valid according to XHTML they will not correctly hook into the DOM tree. Since we are using a numeric tree representation (e.g. 12/0/4/0) one invalid DOM element can mess up the whole tree and we won’t be able to find the elements. While this may sound scary, setting the correct mimetype will fix the issue entirely.

Last reading position 2.0

Eventually we decided to also use this approach to save the user’s last reading position. We now save the first character on a page as the current reading position. We thought back and forth if we might should take a character somewhere from the middle of the page. However with the first character approach we can be sure, that the user always sees something that he has already read, so that he can easily find the last read sentence.

What’s next?

Well, the hard part is – of course – to let the user select text (hint: window.getSelection()) and to visually highlight text fragments. But this is something for another blog post… :)

Dear JavaScript pros: This is a blog post for beginners. So don’t shoot me for simplifying things.

Many developers (especially from the Flash / ActionScript community, but not only them) are currently wanting to “learn” HTML5. Some want to focus on websites, others want to build (single-page-) apps. Some want to do shiny advertising’ish projects, others want to focus more on “products”. And many say, that therefore they need to learn jQuery now.

Do not learn jQuery, just use it!

When someone asks you if you’re into jQuery, just confirm. Chances are, that clients/employers won’t mind to ask, since they assume that you do.

My prediction is, that you’ll use 10 jQuery methods max in the beginning. When starting with jQuery, using jQuery is mostly understanding HTML and CSS (you understand HTML and CSS, right?). Anyway, api.jquery.com is your friend. In a year from now, you’ll probably laugh about your jQuery code, but that’s ok for now, you have bigger problems.

How to produce clean code in JavaScript

The more important question is, how to build clean, manageable, understandable and maintainable JavaScript code. And unlike two years ago there are two industry standards now: backbone.js and require.js.

Backbone enables you to write a class-like code structure, that fulfills all the aforementioned criteria. And – most importantly – since it so widespread many people already know it. For me, working with backbone made working in big teams possible (you want to work in big teams, right?). There are many other frameworks out there, lots of them with a strong feature set. However, what I like about Backbone that it does not have a big feature set.

Require on the other side, not only encourages you to write loosely coupled and granular parts of code but also helps you to structure the JS files that hold your code. It makes you create JavaScript modules in a standardized way, so that you can easily plug them together. I know first hand, how hard it is in the beginning to decide how to organize the JavaScript files/folders and how to get them into the application. Require.js is there for exactly that.

So how to learn them then?

While understanding require is quite straight forward (the documentation will do), the learning curve of backbone is harder. Try to avoid the famous ToDo list example. It gives the impression that backbone applications are always big and/or complicated. The truth is backbone can be extremely simple. Start with something waaaaay simpler.

Unlike other coding communities, the JavaScript community is almost entirely based on open source. It basically lives on github. This includes backbone and require. So if you don’t already have a github account – go and get one. And hang out there for a while.

Other things you might want to check on github:
* HTML5 Boilerplate (especially the .htaccess file is very interesting)
* Twitter Bootstrap issues
* Modernizr
* Good list of things to look into

With the never webkit builds on iOS5 and Android ICS finally overflow:scroll works as expected. On iOS you also have the “native” scroll bounce, which is huge IMHO (with bounce I mean the ease when the scrolled viewport reaches the top/end). The app finally feels native, a big step for web apps.

It wasn’t the sluggishness and weak performance of iScroll and scrollability I was mad about, it was the wrong implementation of the bounce physics. I was always able the tell the difference of a native and a phonegap app by checking scroll views.

Get it working

Check example 1 (with iOS5+ device or simulator)

The magic happens with this line.

Without you would still have a touchmove scroll with one finger, however the “native” bounce comes with -webkit-overflow-scrolling:touch. I notices however one thing that differs from the native scroll. When the content to scroll is on top (e.g. you can’t scroll any further up) and try it anyways, the touchmove bubbles up. Try example 1 again (with iOS5+ device or simulator) In this case it bubble to the top window object which then will scroll the browser chrome. While this might be an expected behavior, it still feels not very native. Let’s fix this.

Make it even more native

Check example 2 (with iOS5+ device or simulator)

When you try the same here, you’ll notice that you can drag the scrolled content with the expected physics. This feels, well, simply amazing. I have been working with all the JavaScript scroll implementations for such a long time now and now it’s finally native. Yeah!

So how is it done?

Well, the magic is to include the scroll content into another container div, that has to have the same height as the parent scroll container and also the correct overflow settings set (-webkit-overflow-scrolling doesn’t seem to be needed). Note that the subcontainer has to have the exact same height, so no margin or padding or stuff allowed. Also note, that no JavaScript is required. This is pure css.

Here a video to illustrate the difference between example 1 and 2:

One last thing

Notice that in the previous example you can still scroll the browser chrome? Try it by dragging (touch moving) the grey background. If you have a web app (instead of a website) you might want to disable the scrolling of the window viewport entirely.

Check example 3 (with iOS5+ device or simulator)

Easy.

Listen to the touchmove event on document level and prevent the default behavior. With this setting alone scrolling would be disabled globally, also in the overflow div. That why you need to set another touchmove listener to the scroll div and stop the bubbling. This event then will never reach the document object.

UPDATE – Arg, found an edge case

Or, actually not so edge. Check a variation of the previous example (with iOS5+ device or simulator) where there are not enough item for the div to be scrollable. When you try to scroll you’ll notice, that the browser chrome scrolls. Not good.

Here corrected example 4 (with iOS5+ device or simulator)

However, it adds more markup and more JavaScript. I would appreciate a smarter solution.

… unless you know what you do.

The problem

Since I started programming I have always been fighting with “strange” code comments. Everyone told me I had to document my code. Also I wanted to be a good citizen and make it as easy as possible for my co-workers to understand my code. On the other hand there was always so little time. And writing good comments takes time. Also I understood that when the codes changes I also have to update the comments (even more time consuming). Plus I was not sure what to comment. Every line? Every method? Or a summary of the class in the header?

When I got better at coding I noticed that no one is writing comments. At least not many. However I found out that comments can be very useful when there is a codeline that needs clarification. So I still felt guilty for not commenting my code.

Last year I got my hands on Clean Code by Robert C. Martin. Dear programmers out there, you need to read this book.

The chapter on Code Comments made me totally change my mind about writing comments and feeling guilty for not writing them. Bottom line: The code itself is the best comment, only write comments when it’s really needed, don’t leave commented code lines.

Example 1 – Annoying

Recently I came into a project with some other programmers. I found the following comment styles (different coding styles aside, that is another story). Which do you find best?

public class AbstractModuleMediator extends Mediator implements IMediator {
    /* public variables and consts */

    /* protected and private variables and consts */

    private static const log : Logger = LogContext.getLogger(AbstractModuleMediator);

    /* public functions */

    /**
     * Constructor for the AbstractModuleMediator class
     */
    public function AbstractModuleMediator(name : String, viewComponent : IModule) {
        super(name, viewComponent);
    }
}

public class SomeModule extends AbstractInteractionModule {
    private static const log : Logger = LogContext.getLogger(SomeModule);

    // ✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖
    // VARIABLES
    // ✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖
    private var _vo : SomeVo;

    // ✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖
    // CONSTRUCTOR
    // ✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖
    public function SomeModule() : void {

    };

    // ✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖
    // PUBLIC FUNCTIONS
    // ✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖✖
    override public function create() : void {
    }
}

public class SomeMediator extends AbstractModuleMediator implements IMediator {

    // _____________________________________________________________________________________________________________
    // C O N S T A N T
    // =============================================================================================================

    public static const NAME : String = "SomeMediator/";

    private static const log : Logger = LogContext.getLogger(SomeMediator);

    // _____________________________________________________________________________________________________________
    // V A R I A B L E S
    // =============================================================================================================

    // _____________________________________________________________________________________________________________
    // I N I T I A L I Z E
    // =============================================================================================================

    public function SomeMediator(name : String, module : IModule) {
        super(name, module);
        log.debug("VideoPlayerMediator()");
    }

    // _____________________________________________________________________________________________________________
    // P R I V A T E - F U N C T I O N S
    // =============================================================================================================
}

public class SomeMediator extends AbstractModuleMediator
{
    public static const NAME:String = "SomeMediator/";

    private static const log:Logger = LogContext.getLogger(SomeMediator);

    public function SomeMediator(viewComponent : IModule)
    {
        super(NAME, viewComponent);
    }

    override public function onRegister() : void
    {
    }
}

So what’s the best commented code? It’s #4. Why? Think about when you first enter a class to make some adjustments. You need to understand the code. You need to read the code. Everything thats distracts you from reading code is useless and should be avoided.

We all know (at least we should) the order of static field, public field, private field, constructor, public methods, private methods. No need to repeat that.

Example 2 – Bad

The other thing Martin points out is the danger of uncommented code lines. This one is harder to achieve since it needs you to make a quick code review on all edited classes before you commit something. Take this example:

facade.registerProxy(new ChannelProxy());
facade.registerProxy(new SoundProxy());
//facade.registerProxy(new SoundAssetProxy());
facade.registerProxy(new SubtitleStateProxy());

facade.registerProxy(new SocialShareProxy());

facade.registerProxy(new TrackProxy());

I had a ticket where I needed to fix a sound issue. I came across this command and the uncommented line left nothing but questions for me. Is this work in progress? Or a permanent removal? Does this Proxy even exist any longer? And most importantly: Does this has something to do with my issue?

Of course someone just forgot to remove this line/comment. However, it cost me some minutes to figure that out.

Example 3 – EVIL

/**
 * Always returns true.
 */
public boolean isAvailable() {
    return false;
}

Do I need to say more? This – of course – is the worst thing that can happen with code comments. The code changes but the comment stays the same. When you change a commented line/function/method and you can – for whatever reason – not change the comment, delete the comment.

Conclusion

As you see from the examples it’s way harder to write good comments then no comments at all. So instead of making it all worse just don’t do it. If you consider yourself as someone who writes exceptable code, that should explain enough for you and your co-workers.

Thats all folks. Oh wait…

//
// Dear maintainer:
//
// Once you are done trying to 'optimize' this routine,
// and have realized what a terrible mistake that was,
// please increment the following counter as a warning
// to the next guy:
//
// total_hours_wasted_here = 39
//

source

stop(); // Hammertime!

source

// I dedicate all this code, all my work, to my wife, Darlene, who will
// have to support me and our three children and the dog once it gets
// released into the public.

source

Note: Some of the points will maybe auto set, when using the final FDT4 Standalone version.

1. Always launch previous application

When hitting F11 by default Eclipse is launching the currently active class (the one you’re currently editing). Of course you want to launch the previous build. Got to Preferences –> Run/Debug –> Launching and set the “Launch Operation” to “Always launch the previous launched application”.

2. Disable Autobuild

When working with big project FDT4 has a hell of work to do when (re-)building the project. This can be seriously annoying. So instead of letting FDT autobuild the project when changes happen, you want to do this manually (ctrl/cmd + B). Goto Project and disable “Build automatically”.

3. Set class comment header

With that you not only have your copyright in the class (not so useful) but also will the svn tag be updated. Go to Preferences –> FDT –> Code Style –> Code Templates –> “typecomment” –> “Edit…” and insert your custom class comment.

4. Set the braces correctly

It drives me nuts, when the braces formating style is not correct ;). Go to Preferences –> FDT –> Code Style –> ActionScript Formatter –> Braces Tab –> “Wrap All”. Clean that mess with ctrl/cmd + shift + F.

5. Make the code assist work more responsive

You want to code assistant to appear when typing and not only when typing . or hitting ctrl/cmd + SPACE. Go to Preferences –> FDT –> Editor –> Code Assist. Set the delay in the “Auto activation Delay:” to 50. In the field “Auto activation triggers for AS:” you insert “abcdefghijklmnopqrstuvwxyz_.:”.

6. Show line numbers

You need then, don’t you? Right click on the grey vertical bar, left to the code editing window and select “Show line numbers”.

7. View classes hierachical in the Flash Explorer

No need to share the space with all classes.

Wait! Are there more?

So these are my 7 steps to get FDT4 up and running. Take it or leave it… :)

Do you have some FDT essentials you can’t live without? Please write a comment!

Some time ago I wrote a blog post about fixing slow iTunes backup. The article became the by far most visited one on this blog and we are still getting tons of visitors from google every day. When reading the comments you can see that lots of users seem to have the same/similar problem.

The issue results from the fact, that some apps save lots of files to the file system of the iPod Touch, iPhone or iPad. While – of course – nothing speaks against using the file system to save data, the problem starts when files getting to many. iTunes then needs ages to copy them to the computer when doing a sync (the backup before the sync in particular).

If using a reasonable number of files you will never have a problem. However, if your app syncs data from the web (for instance a news or rss reader) make sure that you delete the read / no longer needed files. Please :)

The usual customer otherwise might have the feeling the iPhone “doesn’t work” anymore. Or even worse, the customer reads our article and blames your app in the comments… :)

PS: Personally, I haven’t worked on apps that make extensive use of the file system. If you have, feel free to share your experiences.