github.com

classSqliteTokenizer

packages/sql-faker/src/Sqlite/Generation/Tokenization/SqliteTokenizer.php:21
final class SqliteTokenizer
Restricted visibility: declared "@visibility root". Code outside that scope must not name this declaration.

Reads SQLite text back into the parser tokens the server would produce.

This is the inverse of realization and the reason realization can be trusted: SQL written from a terminal sequence is read back here, and the two sequences must agree. SQLite's rules are its own — an identifier may be quoted four ways, a digit-separated number is a token of its own, and a parameter may be written with any of five sigils — so a tokenizer that merely looked plausible would agree with a generator that was equally wrong.

Methods§

public function __construct(private array<string, string> $keywordTokens)
source§

Parameters

$keywordTokensarray<string, string>Terminal name by upper-cased keyword
public function tokenize(string $sql): list<string>
source§

Reads SQL text into the tokens SQLite's own lexer would produce.

Parameters

$sqlstringText to read

Returns

list<string> Parser token names, in order

Throws

LexicalException When the text holds something the lexer cannot read
Test cases 3
Called from 1
Calls 5
public function tokenAt(string $sql, int &$offset): string|null
source§

Reads the one token that starts at the offset, advancing past it.

Parameters

$sqlstringText being read
$offsetintWhere to read from; moved past what was consumed

Returns

string|null The token name, or null when nothing here is a token

Throws

LexicalException When a quoted or bracketed run never closes
Test cases 1
Called from 1
Calls 4
public function quotedTokenAt(string $sql, int &$offset): string|null
source§

Reads a quoted string or a quoted identifier in any of SQLite's spellings.

Parameters

$sqlstringText being read
$offsetintWhere to read from; moved past what was consumed

Returns

string|null The token name, or null when no quoted run opens here

Throws

LexicalException When the run never closes
Test cases 1
Called from 1
Calls 3
public function literalTokenAt(string $sql, int &$offset): string|null
source§

Reads a blob, a bound parameter, or a numeric literal.

Parameters

$sqlstringText being read
$offsetintWhere to read from; moved past what was consumed

Returns

string|null The token name, or null when none of these start here
Test cases 1
Called from 1
Calls 3
public function wordTokenAt(string $sql, int &$offset): string|null
source§

Reads a keyword or an unquoted identifier.

Parameters

$sqlstringText being read
$offsetintWhere to read from; moved past what was consumed

Returns

string|null The token name, or null when no word starts here
Test cases 1
Called from 1
Calls 3
public function operatorTokenAt(string $sql, int &$offset): string|null
source§

Reads an operator or a single punctuation character.

Parameters

$sqlstringText being read
$offsetintWhere to read from; moved past what was consumed

Returns

string|null The token name, or null when nothing operator-like starts here
Test cases 1
Called from 1
Calls 2
public function skipTrivia(string $sql, int &$offset): bool
source§

Consumes whitespace or one comment.

Parameters

$sqlstringText being read
$offsetintWhere to read from; moved past what was consumed

Returns

bool True when something was skipped

Throws

LexicalException When a block comment never closes
Test cases 2
Called from 1
Calls 5
public function skipQuoted(string $sql, int &$offset, string $quote): void
source§

Consumes a quoted run, doubling being the way a quote escapes itself.

Parameters

$sqlstringText being read
$offsetintOffset of the opening quote; moved past the closing one
$quotestringThe quote character

Throws

LexicalException When the run never closes
Test cases 2
Called from 2
Calls 2
public function operatorAt(string $sql, int $offset): array{string, string}|null
source§

Reports the operator written at the offset and the token it stands for.

Parameters

$sqlstringText being read
$offsetintWhere to look

Returns

array{string, string}|null The operator and its token, or null when there is none
Test cases 2
Called from 1
Calls 4
public function normalizedSourceTokens(list<string> $tokens): list<string>
source§

Strips the TK_ prefix and the spacing tokens SQLite's own lexer emits.

Witnesses record what SQLite's lexer produced, which names tokens with a prefix and reports whitespace as a token of its own. Neither belongs in the parser-token sequence this tokenizer is compared against.

Parameters

$tokenslist<string>Tokens as the upstream lexer named them

Returns

list<string> The same tokens as the parser names them
Test cases 2
Calls 2

Private surface 3§

Implementation details, listed for orientation only.

private const OPERATORS = ['->>' => 'PTR', '->' => 'PTR', '||' => 'CONCAT', '==' => 'EQ', '<=' => 'LE', '…
private const PUNCTUATION = ['(' => 'LP', ')' => 'RP', ';' => 'SEMI', ',' => 'COMMA', '.' => 'DOT', '=' => …
private array<string, string> $keywordTokens

Test cases 22§

Test cases that cover or call this symbol, from the coverage report and from the analyzed test sources.

Dedicated tests 16
Other tests reaching this symbol 6

Relations§

Instantiated in 1
Method calls 1
Type declarations 1