feat(compiler): support tokenizing escaped strings (#28055)

In order to support source mapping of templates, we need
to be able to tokenize the template in its original context.
When the template is defined inline as a JavaScript string
in a TS/JS source file, the tokenizer must be able to handle
string escape sequences, such as `\n` and `\"` as they
appear in the original source file.

This commit teaches the lexer how to unescape these
sequences, but only when the `escapedString` option is
set to true.  Otherwise there is no change to the tokenizing
behaviour.

PR Close #28055
This commit is contained in:
Pete Bacon Darwin
2019-02-08 22:10:19 +00:00
committed by Misko Hevery
parent eeb560ac88
commit 2424184d42
4 changed files with 463 additions and 4 deletions

View File

@ -1580,6 +1580,30 @@ export interface ParseTemplateOptions {
* The entire `source` string is parsed if this is not provided.
* */
range?: LexerRange;
/**
* If this text is stored in a JavaScript string, then we have to deal with escape sequences.
*
* **Example 1:**
*
* ```
* "abc\"def\nghi"
* ```
*
* - The `\"` must be converted to `"`.
* - The `\n` must be converted to a new line character in a token,
* but it should not increment the current line for source mapping.
*
* **Example 2:**
*
* ```
* "abc\
* def"
* ```
*
* The line continuation (`\` followed by a newline) should be removed from a token
* but the new line should increment the current line for source mapping.
*/
escapedString?: boolean;
}
/**