Kal Sugar

This module applies a couple of “syntactic sugar” pre-processing steps to Kal code before it goes to the compiler. These steps would be onerous to do during the parsing stage, but are generally easier to do on a token stream. Each function in this module takes an input token stream and returns a new, possibly modified one.

Some sugar functions use the keyword list from the grammar, most notable the implicit parentheses for function calls.

The entry point for this module is the translate_sugar function, which takes an input token stream and returns a modified token stream for use with the parser. It also takes an optional options parameter which may contain the following properties:

• show_tokens - if true, this module will print the input token stream to the console. This is useful for debugging the compiler

The function also takes a tokenizer argument which is a function that given a code string, returns an array with the first element being a token array and the second a comment token array. The Kal compiler uses the tokenize funtion in the lexer module for this argument. tokenizer, if present, is used to tokenize code embedded in double-quoted strings. If this argument is missing, double-quoted strings with embedded code blocks will be left as strings.

The current sugar stages are:

1. code_in_strings - for double-quoted strings with embedded code blocks ("1 + 1 = #{1 + 1}"), this function tokenizes the code blocks and converts the string to the equivalent of "1 + 1 = " + (1 + 1).
2. clean - removes whitespace
3. multiline_statements - removes line breaks after commas on long statements
4. multiline_lists - this function collapses list definitions that span muliple lines into a single line, though the tokens do still retain their original line numbers.
5. no_paren_function_calls - adds parentheses around implicit function calls like my_function 1, 2.
6. print_statement - converts calls to print to console.log
7. coffee_style_functions - converts functions with CoffeeScript syntax ((a,b) -> return a + b) to standard Kal function syntax.

The output is a new token stream (array).

Debug printing of the token stream is enabled with the show_tokens option.

Code In Strings

This function allows support for double-quoted strings with embedded code, like: “x is #{x}”. It uses the tokenizer argument (a function that converts a code string into a token array, like lexer.tokenize) to run the code blocks in the string through the lexer. The return value is the merged stream of tokens.

We abort if there is no tokenizer provided and just don't translate the strings.

The output is a new token array (we don't modify the original).

For double-quoted strings, we search for code blocks like "#{code}". The regex uses the non-greedy operator to avoid parsing "#{block1} #{block2}" as a single block.

We generally must add parentheses around any string that gets broken up for code blocks (and it is always safe to do so). soft indicates that this was added by the sugar module, not the user. It's passed forward to no-paren function calls.

For each code block match, we first add a string token to the stream for all the constant text before the block start, then a +.

Next we add the parsed version of the code block (a token array) generated by running the code through the lexer. If there is more than one token, this also needs to be in parentheses.

Next we make a string out of any remaining text after the block in case this is the last match. If the loop exits here, it gets added to the token stream, otherwise we ignore it since the next iteration will take care of it. If the string is the empty string, we set it to blank since we don't want things like "a is #{a}" turning into ("a is " + a + "") for asthetic reasons.

Find the next code block if there is one.

If there wasn't a next code block, add the remaining string (if any) and close paren.

For anything other than a double-quoted string, just pass it through.

Clean

Removes whitespace. It marks tokens that were followed by whitespace so that the later stages can detect the difference between things like my_function(a) -> and my_function (a) ->.

Multiline Statements

This function removes newlines and indentation after commas, allowing long lines of code to be broken up into multiple lines. Token line numbers are preserved for error reporting.

We keep track of whether or not we are on a continued line and how many indents we ignored.

If we see a newline after a comma, remove it from the stream and mark that we are in line continuation mode.

In line continuation mode, ignore indents and dedents, but keep track of them. We exit line continuation mode when we see a DEDENT that brings back to even with the original line.

When exiting line continuation mode, we have to add back in the last NEWLINE.

Add the token to the new stream unless we decided to skip it.

No-Paren Function Calls

This stage converts implicit function calls (my_function a, b) to explicit ones (my_function(a,b)). NOPAREN_WORDS specify keywords that should not be considered as a first argument to a function call. For example, we don't want x is a to turn into x(is(a)), but we do want x y z to become x(y(z)).

This function is admittedly messy and in need of a rewrite. But it's not broken, so…

We need a token counter because sometimes we look back two or three tokens.

Check that the previous token is not a reserved word. This can happen if the last token is not a keyword, two tokens ago was a . (like x.for a), or the last token is a keyword but a valid r-value (me x).

Check if the previous token was callable. This is only true if it is an IDENTIFIER (not reserved) or a ] like x[1] a.

Check that the current token isn't a no-paren word (not looking at something like x for).

Check that the current token is not a literal (don't want my_function * 2 to become my_function(* 2)).

There are some exceptions for callable literals, for things like f {x:1}, f [1], and ->.

Combining previous checks, we check that this token is not an operator.

Check if this is a function declaration.

Check if a parenthesis is soft, meaning added by the sugar and not the user.

Don't want to add parentheses around bitwise left or bitwise right, but we also really don't want left and right to be no-paren words, otherwise x left would not translate to x(left). These are really useful words, so we handle them in this special case to avoid this issue.

If the previous token is callable and the current token is not an operator (or it‘s a parenthesis that the user didn’t add) and we're not in the special bitwise case, then we add an open paren. We add a trigger to close the parentheses on the next NEWLINE.

If we're passing a function as an argument, we want to change the close trigger to a DEDENT and ignore the next INDENT.

Keep track of indents so that streams like: x = myfunct function () NEWLINE INDENT ... DEDENT will not close out parentheses early.

Reset the ignore_next_indent flag if necessary.

Check if we hit a “closure” (end of implied parentheses) when we are looking for a NEWLINE. This can happen on an actual NEWLINE or when we hit a tail conditional.

If so, pop all NEWLINE closures and add in the implied tokens. NEWLINEs can close out multiple parentheses (x = a b c).

If our closure had a DEDENT trigger, pop it and add the token.

If no trigger was matched, just pass the token through.

If we hit EOF, pop out all the remaning closures.

Coffee-Style Functions

This function converts CoffeeScript-style functions (() ->) to Kal syntax.

We need to track the token index since we look back several tokens in this stage.

Look for a ->.

If we see the ->, that means the current token is > and we already added the - to the new stream. We have to pop the - off the stream.

We create a new token stream fragment for this function header.

Next we examine the last token in the stream. Since we just popped the -, this will either be a ) if the definition is in the form (args) -> or something else if it doesn't specify arguments.

If there are arguments here, keep popping until we hit the (, adding the argument tokens to the new_tokens stream. At the end of this loop, new_tokens will be the arguments passed (if any) without enclosing parens.

Pass the closing paren.

If no arguments were specified, let new_tokens be ()

Prepend the function token to new_tokens, which currently has the arguments (if any) in parentheses. Then add it to the out_tokens stream.

If we're not handling a Coffee-Style function, just pass tokens through.

Multiline Lists

This function converts list definitions that span multiple lines into a single line. Tokens retain their original line numbers. This supports lists and explicit map definitions ({}).

This function is admittedly awful and needs rework.

We need to track nested lists.

We need to keep track of whether or not this token is eligible as a list item separator.

When we see a list start, we push to the list stack.

Likewise for a list end, we pop the stack.

Keep track of the indentation level, looking for a token that returns us to the original indent. We continue to skip indents/dedents until this happens. Basically, we want to ignore indentation inside these multi-line definitions. Once back to original the indent level, we push in a NEWLINE.

Note that none of this happens unless we are inside a list definition (all these flags are ignored).

Skip newlines inside of list definitions.

The first token in a newline stretch gets turned into a comma

Print Statements

Convert print tokens to console . log tokens.