Class Tokenizer

constructor

• new Tokenizer(rules: any): Tokenizer

• Constructs a new tokenizer based on the given rules and flags.

  Parameters

  • rules: any

    The highlighting rules

  Returns Tokenizer

matchMappings

regExps

reportError

splitRegex

states

createSplitterRegexp

• createSplitterRegexp(src: string, flag: string): RegExp

• Parameters

  • src: string
  • flag: string

  Returns RegExp

getLineTokens

• getLineTokens(
      line: string,
      startState: string | string[],
  ): { state: string | string[]; tokens: Token[] }

• Returns an object containing two properties: tokens, which contains all the tokens; and state, the current state.

  Parameters

  • line: string
  • startState: string | string[]

  Returns { state: string | string[]; tokens: Token[] }

removeCapturingGroups

• removeCapturingGroups(src: string): string

• Parameters

  • src: string

  Returns string