clickhouse: type inference via the new analysis core (analyze only) - #4521
Merged
Conversation
kyleconroy
marked this pull request as ready for review
July 27, 2026 02:46
kyleconroy
force-pushed
the
claude/clickhouse-xqlc-core
branch
from
July 28, 2026 02:42
3e6640c to
59e579d
Compare
kyleconroy
commented
Jul 28, 2026
kyleconroy
force-pushed
the
claude/clickhouse-xqlc-core
branch
from
July 28, 2026 20:29
4f43670 to
13790dc
Compare
Port xqlc's core catalog (SQLite-backed sql_* catalog) and its dialect-neutral query analyzer into internal/core, repointing the analyzer from xqlc's copy of the AST onto sqlc's internal/sql/ast so there is a single AST. No converter and no second AST package. A smoke test drives the analyzer with sqlc's own PostgreSQL parser to prove the repointed analyzer resolves columns, types, star expansion, and aliases end-to-end against internal/sql/ast. This is the first step of merging xqlc back into sqlc as the future analysis core; ClickHouse will be the first engine wired onto it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XTGxNHW6v1S1YyC9FDSgrK
Wire ClickHouse onto the merged core: a dialect seed registering the built-in ClickHouse types, and a DDL handler that populates the core catalog from CREATE TABLE using sqlc's existing ClickHouse parser. A smoke test proves the full vertical path — ClickHouse SQL -> sqlc's ClickHouse parser -> internal/sql/ast -> core catalog + analyzer -> PrepareResult — resolving column names, types, nullability, source bindings, and star expansion, with none of the legacy compiler analyze step involved. Also fix the ClickHouse converter to render nested type parameters (the inner type of Nullable(T)/Array(T), Decimal precision, etc.) into TypeName.Name instead of dropping them as TODO nodes, so wrapped types resolve to their effective scalar type. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XTGxNHW6v1S1YyC9FDSgrK
Add the EngineClickHouse engine and a dedicated compile path so `sqlc generate` produces Go for ClickHouse entirely through the xqlc core, bypassing the legacy compiler analyze step and the in-memory sql/catalog: - config: add the "clickhouse" engine constant. - compiler: NewCompiler builds a core.Catalog seeded with the ClickHouse dialect; parseCatalog applies schema DDL to it; a new parseQueryCore resolves each query's columns and parameters via core/analyzer and assembles *compiler.Query, reusing only the shared query-metadata parsing. The legacy analyzeQuery/inferQuery/outputColumns path and the analyzer.Analyzer seam are never entered. - codegen: project the core catalog into plugin.Catalog for model/enum generation, and add a ClickHouse -> Go type map (Nullable(T) -> *T, the integer ladder, Float32/64, String, DateTime -> time.Time, ...). - clickhouse parser: compute statement byte-spans with a running offset and a semicolon scan (doubleclick reports statement starts but not ends), so leading "-- name:" annotations fall inside each statement. An endtoend case (clickhouse_select) exercises the full pipeline and its golden Go output is committed. Updating parse_basic/clickhouse's golden reflects the corrected statement spans and now-detected query name/cmd. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XTGxNHW6v1S1YyC9FDSgrK
Introduce internal/core/catalogdb, a sqlc-generated SQLite query layer over the core catalog's own sql_* tables — sqlc analyzing sqlc's catalog. The catalog's schema.sql is the sqlc schema; catalogdb/query.sql holds the queries; //go:generate runs sqlc against them. Generation is offline and the output is committed, so there is no build-time cycle, and because the SQLite engine runs on the legacy compiler (not the core catalog) there is no analysis recursion. Vertical slice: types.go (CreateType/TypeOID/TypeName/LookupType) now delegates to the generated Queries. FindProcs is split into two generated queries — FindProcsAnyNamespace and FindProcsInNamespaces (sqlc.slice) — with Go choosing based on whether namespaces were supplied, replacing the hand-built IN-list. Remaining core files still use raw SQL and will be swept over incrementally. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XTGxNHW6v1S1YyC9FDSgrK
Sweep the remaining hand-written SQL against the sql_* catalog tables onto the generated catalogdb layer, completing the conversion started in the types.go/FindProcs slice. Namespace, dialect, type, class, attribute, constraint, proc, operator, and cast queries now all run through catalogdb.Queries. Notable rewrites: - FindOperators: the conditionally-appended type filters become a single static query using (@arg = 0 OR col = @arg) sentinels. - Nullable OID columns round-trip through sql.NullInt64 (orZero maps NULL to the 0 sentinel the Go API uses). - New typed catalog accessors back the cross-package call sites that previously reached into the raw handle: DropClass (ClickHouse DROP TABLE), ClassColumns (analyzer scope), and Namespaces / TablesInNamespace / ClassCodegenColumns (codegen catalog projection). Only the schema bootstrap (db.Exec(ddl)) and the DB()/Close() handles remain as raw database/sql; every catalog query is now sqlc-generated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XTGxNHW6v1S1YyC9FDSgrK
Extract schema.sql and query.sql from internal/core (and catalogdb) into a dedicated internal/core/catalogdef package that owns the SQL defining sqlc's catalog and embeds the schema (catalogdef.Schema). sqlc reads its schema and queries from catalogdef and still generates the querier into catalogdb, keeping SQL sources separate from generated code. core/catalog.go now builds its in-memory catalog from catalogdef.Schema instead of embedding schema.sql itself, so the DDL used at runtime and the schema fed to codegen are one source of truth. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XTGxNHW6v1S1YyC9FDSgrK
Remove comments from the Go files authored for this work (catalog core, analyzer, ClickHouse engine, compile path, codegen type map) and drop the comments added to the files touched along the way, keeping //go:generate and //go:embed directives. No remaining references to xqlc. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XTGxNHW6v1S1YyC9FDSgrK
Move pluginCatalogFromCore into a new internal/core/shim package as PluginCatalog. Move the engine-neutral DDL apply into a new internal/core/schema package and have the compiler call it directly, so no clickhouse-specific code remains in the schema loading loop. Push the ClickHouse type unwrapping into the engine's column conversion so the core schema apply only sees canonical scalar types. Rename the core compile path file to parse_core.go. Remove the unit tests in favor of end-to-end coverage.
Wire the ClickHouse engine into `sqlc analyze` so the core catalog and analyzer can be exercised end to end without committing to code generation yet. The analyze command builds the compiler, applies the schema DDL to the core catalog, and reports inferred result columns as JSON — the same static-analysis path `generate` would use. Remove the not-yet-ready code generation surface: the ClickHouse Go type map, the core-catalog to plugin.Catalog projection, and the generate golden test. ClickHouse type inference is now covered by an analyze_basic end-to-end case, matching how the other engines are tested.
kyleconroy
force-pushed
the
claude/clickhouse-xqlc-core
branch
from
July 30, 2026 18:21
acff325 to
2ca774c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
sqlc already has a working ClickHouse parser (
internal/engine/clickhouse, backed bydoubleclick), but no analysis for it: the ClickHouse catalog was a stub, the engine was not wired into the compiler, and there was noEngineClickHouse. The missing layer is a typed catalog + a query analyzer.This PR lands a new analysis core in sqlc and makes ClickHouse the first engine to run on it. ClickHouse is a natural first target because it has a parser and zero existing analysis — this is net-new, with no regression surface for the other engines.
Scope is deliberately limited to
sqlc analyze. There is no code generation for ClickHouse in this PR: the goal is to exercise the catalog and analyzer end to end, and prove the type inference, before committing to a generated-code surface. This follows the same shape as #4522, which wires GoogleSQL intoanalyzeonly.What's here
sqlc analyze --dialect clickhouseresolves result column types entirely through the new core.$ sqlc analyze --dialect clickhouse --schema schema.sql query.sqlGiven
id UInt64, name String, tag Nullable(String), amount Float64, created DateTime:[ { "name": "ListEvents", "cmd": ":many", "columns": [ { "name": "id", "data_type": "uint64", "not_null": true, "table": "events" }, { "name": "name", "data_type": "string", "not_null": true, "table": "events" }, { "name": "tag", "data_type": "string", "not_null": false, "table": "events" }, { "name": "amount", "data_type": "float64", "not_null": true, "table": "events" }, { "name": "created", "data_type": "datetime", "not_null": true, "table": "events" } ], "params": [] } ]Note
tag:Nullable(String)correctly unwraps to a nullablestring.The core (
internal/core/)sql_*tables are modeled on the Postgres system catalogs (namespaces, classes, attributes, types, casts, operators, procs).internal/core/analyzer/— a dialect-neutral analyzer that resolves projections and expression types against that catalog.internal/core/schema/— engine-neutral DDL application (CREATE TABLE/DROP TABLE→ catalog).sqlc builds sqlc. Every query the catalog runs against its own
sql_*tables is generated by sqlc's SQLite engine frominternal/core/catalogdef/{schema.sql,query.sql}intointernal/core/catalogdb, viago:generate. It is idempotent, offline, and introduces no build-time cycle — the SQLite engine uses the legacy compiler, so there is no analysis recursion.ClickHouse
seed.go— the ClickHouse dialect and its built-in types.Nullable(String),Array(UInt64),Decimal(18,4)) instead of dropping them asTODO, and unwraps them to a canonical scalar plus array/nullable flags so the core catalog only ever sees scalars.-- name:annotations fall inside each statement.Compiler
config: theclickhouseengine constant.NewCompilerbuilds a core catalog seeded with the ClickHouse dialect;parseCatalogapplies schema DDL to it;parse_core.goresolves columns viacore/analyzerand assembles*compiler.Query, reusing only the shared query-metadata parsing.Tests —
analyze_basic/clickhousecovers the pipeline end to end, matching how the other engines are tested. Per review feedback, there are no unit tests; coverage is end-to-end only.Design decisions
internal/sql/ast; the analyzer was written against it directly, not a second copy with a converter.internal/sql/catalogis not used on this path.analyzeQuery/inferQuery/outputColumnsor theanalyzer.Analyzerinterface.compilerbeyond parser construction, which matches how the other engines are wired.Follow-ups
WHEREclauses and bind parameters. The analyzer currently handles projectingSELECTs;SELECT ... WHERE id = {id:UInt64}still reportsa_expr: unsupported kind 0, so theanalyzecase is scoped to what actually works.SELECT(INSERT/UPDATE/DELETE).plugin.Catalogprojection and a ClickHouse → Go type map. Both existed in an earlier revision of this branch and were removed to keep this PR analyze-only; richer types (decimal.Decimal,uuid.UUID,netip.Addr,json.RawMessage) need import wiring.testgen-style conformance suite against a real ClickHouse.This is the first engine on the new core. When other engines migrate, they follow the same path and the legacy analyze machinery can be retired.
go build ./...andgo vetpass;go generate ./internal/core/...is idempotent; theanalyze_basic/clickhouseandparse_basic/clickhouseend-to-end cases are green.🤖 Generated with Claude Code
https://claude.ai/code/session_01XTGxNHW6v1S1YyC9FDSgrK