RubyGems Navigation menu

rubylexer 0.7.7

RubyLexer is a lexer library for Ruby, written in Ruby. Rubylexer is meant as a lexer for Ruby that's complete and correct; all legal Ruby code should be lexed correctly by RubyLexer as well. Just enough parsing capability is included to give RubyLexer enough context to tokenize correctly in all cases. (This turned out to be more parsing than I had thought or wanted to take on at first.) RubyLexer handles the hard things like complicated strings, the ambiguous nature of some punctuation characters and keywords in ruby, and distinguishing methods and local variables. RubyLexer is not particularly clean code. As I progressed in writing this, I've learned a little about how these things are supposed to be done; the lexer is not supposed to have any state of it's own, instead it gets whatever it needs to know from the parser. As a stand-alone lexer, Rubylexer maintains quite a lot of state. Every instance variable in the RubyLexer class is some sort of lexer state. Most of the complication and ugly code in RubyLexer is in maintaining or using this state.


  1. 0.7.7 - January 3, 2010 (332 KB)
  2. 0.7.6 - July 6, 2009 (330 KB)
  3. 0.7.5 - May 23, 2009 (329 KB)
  4. 0.7.4 - May 22, 2009 (329 KB)
  5. 0.7.3 - April 30, 2009 (273 KB)
Show all versions (9 total)

Runtime Dependencies:


  • Caleb Clausen



SHA 256 checksum:


Total downloads 17,904

For this version 3,351

= Copy to clipboard Copied!




Required Ruby Version: None