github.com

classPgTokenizer

packages/sql-faker/src/PostgreSql/Generation/Tokenization/PgTokenizer.php:22
final class PgTokenizer
Restricted visibility: declared "@visibility root". Code outside that scope must not name this declaration.

Reads PostgreSQL text back into the parser tokens the server would produce.

This is the inverse of realization and the reason realization can be trusted: SQL written from a terminal sequence is read back here, and the two sequences must agree. PostgreSQL's rules are its own — a string may be spelled six ways, block comments nest, and an operator is the longest run of operator characters that does not start a comment — so a tokenizer that merely looked plausible would agree with a generator that was equally wrong.

Methods§

public function __construct(
    private array<string, string> $keywordTokens,
    PgLookahead $lookahead,
)
source§

Parameters

$keywordTokensarray<string, string>Terminal name by upper-cased keyword
$lookaheadPgLookaheadSubstitutions the parser frontend makes
public function tokenize(string $sql): list<string>
source§

Reads SQL text into the tokens PostgreSQL's own lexer would produce.

Parameters

$sqlstringText to read

Returns

list<string> Parser token names, in order

Throws

LexicalException When the text holds something the lexer cannot read
Test cases 4
Called from 1
Calls 6
public function tokenAt(string $sql, int &$offset): string|null
source§

Reads the one token that starts at the offset, advancing past it.

Parameters

$sqlstringText being read
$offsetintWhere to read from; moved past what was consumed

Returns

string|null The token name, or null when nothing here is a token

Throws

LexicalException When a quoted or dollar-quoted run never closes
Test cases 1
Called from 1
Calls 5
public function quotedTokenAt(string $sql, int &$offset): string|null
source§

Reads a quoted identifier or string in any of the prefixes PostgreSQL accepts.

Parameters

$sqlstringText being read
$offsetintWhere to read from; moved past what was consumed

Returns

string|null The token name, or null when no quoted run opens here

Throws

LexicalException When the run never closes
Test cases 1
Called from 1
Calls 2
public function dollarTokenAt(string $sql, int &$offset): string|null
source§

Reads a dollar-quoted string or a positional parameter.

Parameters

$sqlstringText being read
$offsetintWhere to read from; moved past what was consumed

Returns

string|null The token name, or null when neither starts here

Throws

LexicalException When a dollar-quoted string never closes
Test cases 1
Called from 1
Calls 4
public function numericTokenAt(string $sql, int &$offset): string|null
source§

Reads an integer or a float literal.

Parameters

$sqlstringText being read
$offsetintWhere to read from; moved past what was consumed

Returns

string|null The token name, or null when no number starts here
Test cases 1
Called from 1
Calls 5
public function wordTokenAt(string $sql, int &$offset): string|null
source§

Reads a keyword or an unquoted identifier.

Parameters

$sqlstringText being read
$offsetintWhere to read from; moved past what was consumed

Returns

string|null The token name, or null when no word starts here
Test cases 1
Called from 1
Calls 3
public function operatorTokenAt(string $sql, int &$offset): string|null
source§

Reads an operator or a single punctuation character.

Parameters

$sqlstringText being read
$offsetintWhere to read from; moved past what was consumed

Returns

string|null The token name, or null when nothing operator-like starts here
Test cases 1
Called from 1
Calls 2
public function skipTrivia(string $sql, int &$offset): bool
source§

Consumes whitespace or one comment.

PostgreSQL nests block comments, so the depth is counted rather than the first *​/ being taken as the end.

Parameters

$sqlstringText being read
$offsetintWhere to read from; moved past what was consumed

Returns

bool True when something was skipped

Throws

LexicalException When a block comment never closes
Test cases 2
Called from 1
Calls 5
public function operatorAt(string $sql, int $offset): array{string, string}|null
source§

Reports the operator written at the offset and the token it stands for.

A user-defined operator is the longest run of operator characters, but the run stops where a comment would start: +/* is a plus and a comment, not a three-character operator.

Parameters

$sqlstringText being read
$offsetintWhere to look

Returns

array{string, string}|null The operator and its token, or null when there is none
Test cases 3
Called from 1
Calls 10
public function fixedOperator(string $operator): string|null
source§

Reports the token a spelling with a name of its own stands for.

Parameters

$operatorstringOperator as written

Returns

string|null The token name, or null when the operator has none
Test cases 1
Called from 2
public function isPunctuation(string $character): bool
source§

Reports whether a single character is punctuation that stands for itself.

Parameters

$characterstringCharacter to inspect

Returns

bool True for a punctuation character
Test cases 1
Called from 2
Calls 3

Private surface 5§

Implementation details, listed for orientation only.

private const OPERATOR_CHARACTERS = '+-*/<>=~!@#%^&|`?'
private const PUNCTUATION = '%()*+,-./:;<=>[]^'
private const PUNCTUATION_PAIRS = ['::', '..', ':=']
private PgLookahead $lookahead
private array<string, string> $keywordTokens

Test cases 25§

Test cases that cover or call this symbol, from the coverage report and from the analyzed test sources.

Dedicated tests 19
Other tests reaching this symbol 6

Relations§

Instantiated in 1
Method calls 1
Type declarations 1