Starting from scratch

I've completed a more than good-enough tokenizer to be going on with, and in short order.

The beauty of starting from scratch -- re-inventing the wheel, if you like -- is that you learn.  Even if you subsequently pick up a more mature component, you'll be in a much better position to evaluate, and where necessary, debug.  A good analogy is to the experience of driving a car if you've already built or re-built one.  Your mental model is deeper than a regular driver [user].

The challenge is the open-endedness of the task, all the options and decisions.  This is where approaches like test-driven-design/development and (used) design-by-contract (not yet, ran into some Node-related problems) help a lot.  They do a few things: you are obliged to make explicit your assumptions, and the computer tests them for you.  Inevitably, there are mistakes.  Better to fail fast and make steady progress than have the illusion of progress, but in fact be digging yourself into a deeper and deeper tar pit.

The answer to choice paralysis is to proceed, but use TDD or DBC or some-such to rigorously test out your choices, and allow yourself to go back and change.  An over-arching objective of simplicity of design, layering a la SICP, and the discipline to keep refactoring as you go helps too.