classSqliteTokenizer

SqliteGeneration packages/sql-faker/src/Sqlite/Generation/Tokenization/SqliteTokenizer.php:21

final class SqliteTokenizer

Restricted visibility: declared "@visibility root". Code outside that scope must not name this declaration.

Reads SQLite text back into the parser tokens the server would produce.

This is the inverse of realization and the reason realization can be trusted: SQL written from a terminal sequence is read back here, and the two sequences must agree. SQLite's rules are its own — an identifier may be quoted four ways, a digit-separated number is a token of its own, and a parameter may be written with any of five sigils — so a tokenizer that merely looked plausible would agree with a generator that was equally wrong.

Methods§

public function __construct(private array<string, string> $keywordTokens)

100%source §

Parameters

$keywordTokens array<string, string> Terminal name by upper-cased keyword

Test cases 41

public function tokenize(string $sql): list<string>

88%source §

Reads SQL text into the tokens SQLite's own lexer would produce.

Parameters

$sql string Text to read

Returns

list<string> Parser token names, in order

Throws

LexicalException When the text holds something the lexer cannot read

Test cases 29

Called from 1

method-call SqlFaker\Sqlite\Generation\LexicalGrammar::tokenize() packages/sql-faker/src/Sqlite/Generation/LexicalGrammar.php:135

Calls 5

function-call strlen line 74
method-call SqliteTokenizer::skipTrivia() line 79
static-call LexicalException::noProgress() line 81
method-call SqliteTokenizer::tokenAt() line 87
static-call LexicalException::unsupportedInput() line 89

public function tokenAt(string $sql, int &$offset): string|null

100%source §

Reads the one token that starts at the offset, advancing past it.

Parameters

`$sql`	`string`	Text being read
`$offset`	`int`	Where to read from; moved past what was consumed

Returns

string|null The token name, or null when nothing here is a token

Throws

LexicalException When a quoted or bracketed run never closes

Test cases 28

Called from 1

method-call SqlFaker\Sqlite\Generation\Tokenization\SqliteTokenizer::tokenize() packages/sql-faker/src/Sqlite/Generation/Tokenization/SqliteTokenizer.php:87

Calls 4

method-call SqliteTokenizer::quotedTokenAt() line 114
method-call SqliteTokenizer::literalTokenAt() line 115
method-call SqliteTokenizer::wordTokenAt() line 116
method-call SqliteTokenizer::operatorTokenAt() line 117

public function quotedTokenAt(string $sql, int &$offset): string|null

100%source §

Reads a quoted string or a quoted identifier in any of SQLite's spellings.

Parameters

`$sql`	`string`	Text being read
`$offset`	`int`	Where to read from; moved past what was consumed

Returns

string|null The token name, or null when no quoted run opens here

Throws

LexicalException When the run never closes

Test cases 29

Called from 1

method-call SqlFaker\Sqlite\Generation\Tokenization\SqliteTokenizer::tokenAt() packages/sql-faker/src/Sqlite/Generation/Tokenization/SqliteTokenizer.php:114

Calls 3

method-call SqliteTokenizer::skipQuoted() line 135
function-call strpos line 150
static-call LexicalException::unterminatedBracketIdentifier() line 152

public function literalTokenAt(string $sql, int &$offset): string|null

100%source §

Reads a blob, a bound parameter, or a numeric literal.

Parameters

`$sql`	`string`	Text being read
`$offset`	`int`	Where to read from; moved past what was consumed

Returns

string|null The token name, or null when none of these start here

Test cases 24

Called from 1

method-call SqlFaker\Sqlite\Generation\Tokenization\SqliteTokenizer::tokenAt() packages/sql-faker/src/Sqlite/Generation/Tokenization/SqliteTokenizer.php:115

Calls 3

function-call preg_match line 176
function-call strlen line 180
function-call str_contains line 191

public function wordTokenAt(string $sql, int &$offset): string|null

100%source §

Reads a keyword or an unquoted identifier.

Parameters

`$sql`	`string`	Text being read
`$offset`	`int`	Where to read from; moved past what was consumed

Returns

string|null The token name, or null when no word starts here

Test cases 13

Called from 1

method-call SqlFaker\Sqlite\Generation\Tokenization\SqliteTokenizer::tokenAt() packages/sql-faker/src/Sqlite/Generation/Tokenization/SqliteTokenizer.php:116

Calls 3

function-call preg_match line 204
function-call strlen line 208
function-call strtoupper line 210

public function operatorTokenAt(string $sql, int &$offset): string|null

100%source §

Reads an operator or a single punctuation character.

Parameters

`$sql`	`string`	Text being read
`$offset`	`int`	Where to read from; moved past what was consumed

Returns

string|null The token name, or null when nothing operator-like starts here

Test cases 7

Called from 1

method-call SqlFaker\Sqlite\Generation\Tokenization\SqliteTokenizer::tokenAt() packages/sql-faker/src/Sqlite/Generation/Tokenization/SqliteTokenizer.php:117

Calls 2

method-call SqliteTokenizer::operatorAt() line 223
function-call strlen line 228

public function skipTrivia(string $sql, int &$offset): bool

100%source §

Consumes whitespace or one comment.

Parameters

`$sql`	`string`	Text being read
`$offset`	`int`	Where to read from; moved past what was consumed

Returns

bool True when something was skipped

Throws

LexicalException When a block comment never closes

Test cases 29

Called from 1

method-call SqlFaker\Sqlite\Generation\Tokenization\SqliteTokenizer::tokenize() packages/sql-faker/src/Sqlite/Generation/Tokenization/SqliteTokenizer.php:79

Calls 5

function-call preg_match line 245
function-call strlen line 246
function-call substr line 251
function-call strpos line 252
static-call LexicalException::unterminatedBlockComment() line 264

public function skipQuoted(string $sql, int &$offset, string $quote): void

100%source §

Consumes a quoted run, doubling being the way a quote escapes itself.

Parameters

`$sql`	`string`	Text being read
`$offset`	`int`	Offset of the opening quote; moved past the closing one
`$quote`	`string`	The quote character

Throws

LexicalException When the run never closes

Test cases 5

Called from 2

Calls 2

function-call strlen line 282
static-call LexicalException::unterminatedQuotedToken() line 299

public function operatorAt(string $sql, int $offset): array{string, string}|null

100%source §

Reports the operator written at the offset and the token it stands for.

Parameters

`$sql`	`string`	Text being read
`$offset`	`int`	Where to look

Returns

array{string, string}|null The operator and its token, or null when there is none

Test cases 9

Called from 1

method-call SqlFaker\Sqlite\Generation\Tokenization\SqliteTokenizer::operatorTokenAt() packages/sql-faker/src/Sqlite/Generation/Tokenization/SqliteTokenizer.php:223

Calls 4

class-const SqliteTokenizer::OPERATORS() line 312
function-call str_starts_with line 313
function-call substr line 313
class-const SqliteTokenizer::PUNCTUATION() line 318

public function normalizedSourceTokens(list<string> $tokens): list<string>

100%source §

Strips the TK_ prefix and the spacing tokens SQLite's own lexer emits.

Witnesses record what SQLite's lexer produced, which names tokens with a prefix and reports whitespace as a token of its own. Neither belongs in the parser-token sequence this tokenizer is compared against.

Parameters

$tokens list<string> Tokens as the upstream lexer named them

Returns

list<string> The same tokens as the parser names them

Test cases 2

SqliteTokenizerTest::testNormalizedSourceTokensLeavesUnprefixedNamesAlone covers and calls
SqliteTokenizerTest::testNormalizedSourceTokensStripsThePrefixAndTheSpacing covers and calls

Calls 2

function-call str_starts_with line 341
function-call substr line 341

Private surface 3§

Implementation details, listed for orientation only.

private const OPERATORS = ['->>' => 'PTR', '->' => 'PTR', '||' => 'CONCAT', '==' => 'EQ', '<=' => 'LE', '…

private const PUNCTUATION = ['(' => 'LP', ')' => 'RP', ';' => 'SEMI', ',' => 'COMMA', '.' => 'DOT', '=' => …

private array<string, string> $keywordTokens

Test cases 48§

Test cases that cover or call this symbol, from the coverage report and from the analyzed test sources.

Dedicated tests 42

Other tests reaching this symbol 6

Relations§

Instantiated in 1

new SqlFaker\Sqlite\Generation\LexicalGrammar::__construct() packages/sql-faker/src/Sqlite/Generation/LexicalGrammar.php:65

Method calls 1

method-call SqlFaker\Sqlite\Generation\LexicalGrammar::tokenize() packages/sql-faker/src/Sqlite/Generation/LexicalGrammar.php:135

Type declarations 1

type SqlFaker\Sqlite\Generation\LexicalGrammar packages/sql-faker/src/Sqlite/Generation/LexicalGrammar.php:47