Class Tokenizer

This class takes a set of highlighting rules, and creates a tokenizer out of them. For more information, see the wiki on extending highlighters.

Hierarchy (View Summary)

Tokenizer
- Tokenizer

Index

Constructors

Properties

matchMappings regExps reportError splitRegex states

Methods

createSplitterRegexp getLineTokens removeCapturingGroups

Constructors

constructor

new Tokenizer(rules: any): Tokenizer
Constructs a new tokenizer based on the given rules and flags.
Parameters
- rules: any
  The highlighting rules
Returns Tokenizer
Inherited from Tokenizer.constructor
- Defined in src/tokenizer.js:14

Properties

matchMappings

matchMappings: {}

regExps

regExps: {}

reportError

reportError: (msg: any, data: any) => void

splitRegex

splitRegex: RegExp

states

states: any

Methods

createSplitterRegexp

createSplitterRegexp(src: string, flag: string): RegExp
Parameters
- src: string
- flag: string
Returns RegExp
Inherited from Tokenizer.createSplitterRegexp
- Defined in src/tokenizer.js:175

getLineTokens

getLineTokens(line: any, startState: any): any
Returns an object containing two properties: tokens, which contains all the tokens; and state, the current state.
Parameters
- line: any
- startState: any
Returns any
Overrides Tokenizer.getLineTokens
- Defined in src/tokenizer.js:220

removeCapturingGroups

removeCapturingGroups(src: string): string
Parameters
- src: string
Returns string
Inherited from Tokenizer.removeCapturingGroups
- Defined in src/tokenizer.js:163