github.com

classMySqlTokenizer

packages/sql-faker/src/MySql/Generation/Tokenization/MySqlTokenizer.php:18
final class MySqlTokenizer
Restricted visibility: declared "@visibility root". Code outside that scope must not name this declaration.

Reads MySQL text back into the parser tokens the server would produce.

This diagnostic tokenizer supports the declared generated spellings and selected scanner contracts. It is not a substitute for an independent native parser verdict.

Methods§

public function __construct(
    private array<string, string> $symbolTokens,
    private array<string, string> $functionTokens,
    private bool $dollarQuotedStrings,
)
source§

Parameters

$symbolTokensarray<string, string>Terminal name by upper-cased keyword or operator
$functionTokensarray<string, string>Terminal name by upper-cased function name
$dollarQuotedStringsboolWhether the server build accepts `$$ ... $$` strings
public function tokenize(string $sql): list<string>
source§

Reads SQL text into the tokens MySQL's own lexer would produce.

Parameters

$sqlstringText to read

Returns

list<string> Parser token names, in order

Throws

LexicalException When the text holds something the lexer cannot read
Test cases 4
Called from 1
Calls 6
public function tokenAt(string $sql, int &$offset, list<string> $emitted): string|null
source§

Reads the one token that starts at the offset, advancing past it.

Parameters

$sqlstringText being read
$offsetintWhere to read from; moved past what was consumed
$emittedlist<string>Tokens read so far, for the rules that depend on them

Returns

string|null The token name, or null when nothing here is a token

Throws

LexicalException When a quoted or dollar-quoted run never closes
Test cases 1
Called from 1
Calls 5
public function quotedTokenAt(string $sql, int &$offset): string|null
source§

Reads a backtick, single-quoted or dollar-quoted run.

Parameters

$sqlstringText being read
$offsetintWhere to read from; moved past what was consumed

Returns

string|null The token name, or null when no quote opens here

Throws

LexicalException When the run never closes
Test cases 1
Called from 1
Calls 4
public function markerTokenAt(string $sql, int &$offset, list<string> $emitted): string|null
source§

Reads a parameter marker, a user-variable sigil, or the hostname after one.

A hostname is only a hostname because an @ came before it; the same characters anywhere else are an ordinary identifier, which is why the tokens read so far decide this.

Parameters

$sqlstringText being read
$offsetintWhere to read from; moved past what was consumed
$emittedlist<string>Tokens read so far

Returns

string|null The token name, or null when none of these start here
Test cases 2
Called from 1
Calls 3
public function numericTokenAt(string $sql, int &$offset): string|null
source§

Reads a numeric literal in any of the spellings MySQL accepts.

Parameters

$sqlstringText being read
$offsetintWhere to read from; moved past what was consumed

Returns

string|null The token name, or null when no number starts here

Throws

LexicalException When a prefixed quoted literal never closes
Test cases 1
Called from 1
Calls 8
public function wordTokenAt(string $sql, int &$offset): string|null
source§

Reads a keyword, function name, charset introducer or identifier.

Parameters

$sqlstringText being read
$offsetintWhere to read from; moved past what was consumed

Returns

string|null The token name, or null when no word starts here
Test cases 1
Called from 1
Calls 5
public function operatorTokenAt(string $sql, int &$offset): string|null
source§

Reads an operator or a single punctuation character.

Parameters

$sqlstringText being read
$offsetintWhere to read from; moved past what was consumed

Returns

string|null The token name, or null when nothing operator-like starts here
Test cases 1
Called from 1
Calls 2
public function skipTrivia(string $sql, int &$offset): bool
source§

Consumes whitespace or one comment.

Parameters

$sqlstringText being read
$offsetintWhere to read from; moved past what was consumed

Returns

bool True when something was skipped

Throws

LexicalException When a block comment never closes
Test cases 2
Called from 1
Calls 5
public function skipQuoted(string $sql, int &$offset, string $quote): void
source§

Consumes a quoted run, doubling being the way a quote escapes itself.

Parameters

$sqlstringText being read
$offsetintOffset of the opening quote; moved past the closing one
$quotestringThe quote character

Throws

LexicalException When the run never closes
Test cases 2
Called from 3
Calls 2
public function integerToken(string $integer): string
source§

Reports which integer token a run of digits widens to.

MySQL picks the narrowest of three tokens that can hold the value, so the boundaries are compared as text: the digits may be wider than an int.

Parameters

$integerstringThe digits as written

Returns

string One of NUM, LONG_NUM or ULONGLONG_NUM
Test cases 1
Called from 1
Calls 3
public function operatorAt(string $sql, int $offset): string|null
source§

Reports the operator written at the offset, longest first.

Parameters

$sqlstringText being read
$offsetintWhere to look

Returns

string|null The operator as written, or null when there is none
Test cases 2
Called from 1
Calls 4
public function isPunctuation(string $character): bool
source§

Reports whether a single character is punctuation the grammar uses.

Parameters

$characterstringCharacter to inspect

Returns

bool True for a punctuation character
Test cases 1
Called from 1
Calls 3
public function merged(list<string> $tokens): list<string>
source§

Joins the token pairs MySQL's parser sees as one.

Parameters

$tokenslist<string>Tokens as read

Returns

list<string> Tokens with `WITH ROLLUP` collapsed
Test cases 2
Called from 1

Private surface 5§

Implementation details, listed for orientation only.

private const OPERATORS = ['<=>', '->>', '&&', '<=', '<>', '!=', '>=', '<<', '>>', ':=', '->', '||']
private const PUNCTUATION = '!%&()*+,-./:;@^{}|~=<>'
private array<string, string> $symbolTokens
private array<string, string> $functionTokens
private bool $dollarQuotedStrings

Test cases 28§

Test cases that cover or call this symbol, from the coverage report and from the analyzed test sources.

Dedicated tests 21
Other tests reaching this symbol 7

Relations§

Instantiated in 1
Method calls 1
Type declarations 1