classPgTokenizer

PostgreSqlGenerationpackages/sql-faker/src/PostgreSql/Generation/Tokenization/PgTokenizer.php:22
final class PgTokenizer
Restricted visibility: declared "@visibility root". Code outside that scope must not name this declaration.

Reads PostgreSQL text back into the parser tokens the server would produce.

This is the inverse of realization and the reason realization can be trusted: SQL written from a terminal sequence is read back here, and the two sequences must agree. PostgreSQL's rules are its own — a string may be spelled six ways, block comments nest, and an operator is the longest run of operator characters that does not start a comment — so a tokenizer that merely looked plausible would agree with a generator that was equally wrong.

Methods§

public function __construct(
    private array<string, string> $keywordTokens,
    PgLookahead $lookahead,
)
100%source§

Parameters

$keywordTokensarray<string, string>Terminal name by upper-cased keyword
$lookaheadPgLookaheadSubstitutions the parser frontend makes
Test cases 59
public function tokenize(string $sql): list<string>
88%source§

Reads SQL text into the tokens PostgreSQL's own lexer would produce.

Parameters

$sqlstringText to read

Returns

list<string> Parser token names, in order

Throws

LexicalException When the text holds something the lexer cannot read
Test cases 45
Called from 1
Calls 6
public function tokenAt(string $sql, int &$offset): string|null
100%source§

Reads the one token that starts at the offset, advancing past it.

Parameters

$sqlstringText being read
$offsetintWhere to read from; moved past what was consumed

Returns

string|null The token name, or null when nothing here is a token

Throws

LexicalException When a quoted or dollar-quoted run never closes
Test cases 44
Called from 1
Calls 5
public function quotedTokenAt(string $sql, int &$offset): string|null
100%source§

Reads a quoted identifier or string in any of the prefixes PostgreSQL accepts.

Parameters

$sqlstringText being read
$offsetintWhere to read from; moved past what was consumed

Returns

string|null The token name, or null when no quoted run opens here

Throws

LexicalException When the run never closes
Test cases 45
Called from 1
Calls 2
public function dollarTokenAt(string $sql, int &$offset): string|null
100%source§

Reads a dollar-quoted string or a positional parameter.

Parameters

$sqlstringText being read
$offsetintWhere to read from; moved past what was consumed

Returns

string|null The token name, or null when neither starts here

Throws

LexicalException When a dollar-quoted string never closes
Test cases 38
Called from 1
Calls 4
public function numericTokenAt(string $sql, int &$offset): string|null
100%source§

Reads an integer or a float literal.

Parameters

$sqlstringText being read
$offsetintWhere to read from; moved past what was consumed

Returns

string|null The token name, or null when no number starts here
Test cases 34
Called from 1
Calls 5
public function wordTokenAt(string $sql, int &$offset): string|null
100%source§

Reads a keyword or an unquoted identifier.

Parameters

$sqlstringText being read
$offsetintWhere to read from; moved past what was consumed

Returns

string|null The token name, or null when no word starts here
Test cases 26
Called from 1
Calls 3
public function operatorTokenAt(string $sql, int &$offset): string|null
100%source§

Reads an operator or a single punctuation character.

Parameters

$sqlstringText being read
$offsetintWhere to read from; moved past what was consumed

Returns

string|null The token name, or null when nothing operator-like starts here
Test cases 18
Called from 1
Calls 2
public function skipTrivia(string $sql, int &$offset): bool
100%source§

Consumes whitespace or one comment.

PostgreSQL nests block comments, so the depth is counted rather than the first *​/ being taken as the end.

Parameters

$sqlstringText being read
$offsetintWhere to read from; moved past what was consumed

Returns

bool True when something was skipped

Throws

LexicalException When a block comment never closes
Test cases 45
Called from 1
Calls 5
public function skipQuoted(string $sql, int &$offset, string $quote): void
100%source§

Consumes a quoted run, doubling being the way a quote escapes itself.

Parameters

$sqlstringText being read
$offsetintOffset of the opening quote; moved past the closing one
$quotestringThe quote character

Throws

LexicalException When the run never closes
Test cases 9
Called from 3
Calls 2
public function operatorAt(string $sql, int $offset): array{string, string}|null
100%source§

Reports the operator written at the offset and the token it stands for.

A user-defined operator is the longest run of operator characters, but the run stops where a comment would start: +/* is a plus and a comment, not a three-character operator.

Parameters

$sqlstringText being read
$offsetintWhere to look

Returns

array{string, string}|null The operator and its token, or null when there is none
Test cases 21
Called from 1
Calls 10
public function fixedOperator(string $operator): string|null
90%source§

Reports the token a spelling with a name of its own stands for.

Parameters

$operatorstringOperator as written

Returns

string|null The token name, or null when the operator has none
Test cases 15
Called from 2

Private surface 5§

Implementation details, listed for orientation only.

private const OPERATOR_CHARACTERS = '+-*/<>=~!@#%^&|`?'
private const PUNCTUATION = '%()*+,-./:;<=>[]^'
private const PUNCTUATION_PAIRS = ['::', '..', ':=']
private PgLookahead $lookahead
private array<string, string> $keywordTokens

Test cases 66§

Test cases that cover or call this symbol, from the coverage report and from the analyzed test sources.

Dedicated tests 60
Other tests reaching this symbol 6

Relations§

Instantiated in 1
Method calls 1
Type declarations 1