Migrating to dbx¶
This guide moves an application from the legacy db package (db.Repository, db.Grid, goqu,
db/qb, db/field) and the legacy provider/clickhouse repository to
dbx repositories built on gohan. It gives a per-repository checklist, a
mapping for every db.Repository method, the db.Grid → dbx.Grid differences, a goqu → gohan
expression table, the behaviour that changed on purpose, and a complete before/after example.
Existing db.Repository code keeps working; nothing forces the move. db and dbx run side by
side on the same client, so repositories can move one at a time (see
Migrating incrementally).
Why migrate¶
- Values are always bound. Since v0.11.0,
dbrenders goqu datasets with bound values, but agoqu.Lfragment built withfmt.Sprintfis still inlined, and the legacyprovider/clickhouserepository still renders every query with goqu'sToSQL(), which inlines values as escaped literals.gohanbinds every value; onlyRaw's SQL text,Fn's function name andCast's type string are trusted input. - Identifiers are always quoted and escaped.
dbonly rejects column names containing",\or NUL (db.ErrInvalidIdentifier);gohanquotes and escapes every table and column name for the dialect. dbxalso returns typed results (*T,[]*T), takes acontext.Contexton every call, refuses unfilteredUPDATE/DELETE, validates grid input, and writes multi-statement inserts atomically.
Checklist: migrating one repository¶
-
Create the Querier and repository. For PostgreSQL and SQLite, wrap the
*db.SqlClientthedb.Repositoryalready uses. For ClickHouse, use the client'sQuerier():q, err := dbx.FromClient(client) // the same *db.SqlClient db.NewRepository used if err != nil { return err } users, err := dbx.NewRepository[User](q, "users") if err != nil { return err } // ClickHouse: provider/clickhouse's *Client events, err := dbx.NewRepository[Event](chClient.Querier(), "events") if err != nil { return err }FromClientresolves the dialect from the client's driver name (pgx,pgx/v5,postgres,sqlite,sqlite3); any other name fails withgohan.ErrUnknownDialectunless registered withgohan.Register, and theclickhousedriver fails withdbx.ErrDialectDriver. It connects the client if needed and keeps its*sqlx.DB, so call it once at startup. Build oneQuerierand share it between repositories:dbx.WithTxneeds it.NewRepositorychecks the record type up front (see step 2), so run it where a failure stops startup. -
Check the record struct tags.
dbxreads struct metadata throughgohan/field, whose rules are the same asdb/field's except where noted below:Tag How dbx/gohanreads itcolumn name the dbtag's name, else thechtag's name, else the lower-cased Go field name.db:"-"(orch:"-"on a field without adbtag) skips the field.auto db:"name,auto",auto:"true",grid:"auto",goqu:"skipinsert"andgoqu:"skipupdate"all mean the same thing: the column is never inserted and never updated (unlessgohan.WithAutoFields()),Changeset.Setrejects it anddbx.Changesignores it.goqu:"skipupdate"therefore also drops the column fromINSERT, as it already did indb.Repository's own insert and update builders (only goqu's dataset builders,SqlInsert()/SqlUpdate(), treated the two separately).omit only goqu:"omitnil"(skip a nil pointer) andgoqu:"omitempty"(skip a zero value). They apply toInsertand toUpdate'sSetRecord.db:"x,omitnil"sets nothing, as before.other goqu:optionsignored (e.g. defaultifempty).grid grid:"sort,filter,search"; the grid alias is thealiastag, elsejson, elsexml, else the Go field name.mappernot read by db,dbxorgohan. For a JSON column usejsoncol.JSON[T].Fix these while you are there:
- No whitespace in tags.
db/fieldtrims tag parts, sodb:"id, auto"andgrid:"sort, filter"work indb.gohan/fielddoes not:" auto"and" filter"are silently ignored, so the column is inserted and the field is not filterable. Writedb:"id,auto"andgrid:"sort,filter". - Record shapes.
NewRepositoryfails withgohan.ErrRecordShapefor an embedded pointer struct, an unexported ordb/ch-tagged embedded struct, or an ambiguous promoted field, and withgohan.ErrDuplicateColumnwhen two fields map to one column. - Reserved types. A type registered with
db/field.AddReservedTypemust also be registered withgohan/field.AddReservedType; the two registries are separate (see Structs and Tags). - ClickHouse:
chmust matchdb. The legacy ClickHouse repository scanned and inserted through clickhouse-go, which reads only thechtag (falling back to the Go field name as written), while its grid used thedbtag.dbxbuilds column lists from thedbtag, so every mapped field must reach the same column under both rules: give each field achtag equal to itsdbtag, or use onlychtags.NewRepositoryoverclient.Querier()checks this (dbx.RecordChecker) and fails withclickhouse.ErrRecordMapping, naming each offending column. A record likedb:"id" ch:"event_id"is rejected, and so is a field with onlydb:"name"(clickhouse-go maps it asName). See Record types need matchingchtags.
// Before: accepted by db.Repository type Account struct { ID int64 `db:"id, auto"` // " auto" is ignored by gohan Email string `db:"email" grid:"search, filter"` // " filter" is ignored by gohan Bio *string `db:"bio,omitnil"` // not an omit option anywhere Profile string `db:"profile" mapper:"json"` // mapper is never read } // After type Account struct { ID int64 `db:"id,auto"` Email string `db:"email" grid:"search,filter"` Bio *string `db:"bio" goqu:"omitnil"` Profile jsoncol.JSON[Profile] `db:"profile"` } - No whitespace in tags.
-
Replace the calls using the method mapping. Pass
ctxto every call; reads return*T/[]*Tinstead of filling a target. -
Move transactions to
dbx.WithTx. Wrap the work indbx.WithTx(ctx, q, opts, fn)and run every statement insidefnthroughrepo.With(tx). Any repository can join the transaction, not just the table it was begun from. Serializable retry loops becomedbx.WithTxRetry, andset_config/SET LOCAL ROLEpreambles becomepgsql.WithTxSession(see Transactions). -
Replace goqu expressions with gohan ones using the expression table. Start custom
SELECTs forGet/Listfromrepo.Select(), notgohan.From(table), which rendersSELECT *. Usedbx.Query[D]/QueryOne[D]for joins, aggregates and other shapes that are notT. -
Update error handling.
db.EmptyResult(err)→errors.Is(err, dbx.ErrNotFound)(dbx.ErrNotFoundissql.ErrNoRows).Get,GetBy,QueryOneandUpsertReturningreturn it;List,ListBy,Query,QueryGridandUpdateReturningreturn an empty, non-nil slice instead.err.(db.GridError)→var gridErr dbx.GridError; errors.As(err, &gridErr). It is a different type, so an assertion againstdb.GridErrornever matches.- New errors to handle:
gohan.ErrNoWhere(nil or trivially-truewhereonUpdate/UpdateFields/UpdateReturning/Delete),gohan.ErrEmptyMatch(an empty map inGetBy/ListBy/gohan.Match),dbx.ErrUnknownColumn(a map key or column name that is not one ofT's columns),gohan.ErrUnsupported(a feature the dialect lacks, e.g.RETURNINGon ClickHouse),dbx.ErrTxUnsupported(a transaction over a Querier that cannot begin one) andgohan.ErrInconsistentOmit(seeWithGroupedInserts). db.ErrInvalidParameters,db.ErrInvalidIdentifierand thedb/qbvalidation errors are not returned bydbx.
-
Update grid handlers. Decode
dbx.GridQueryinstead ofdb.GridQuery, build the grid once withdbx.NewGrid[T](), and read the grid differences before shipping: the default 1000-row cap and the stricter validation change responses. -
Run the tests.
go test -race ./...(Docker is needed for the testcontainers suites). Cover each migrated repository at least with a test that calls its constructor, soNewRepositoryshape errors and ClickHouseCheckRecorderrors fail the tests, not startup.
db.Repository method mapping¶
repo is a *dbx.Repository[T], q the Querier from dbx.FromClient (a *dbx.SQLQuerier),
where a gohan.Expr. Raw SQL run through q uses the dialect's own placeholders ($1 on
PostgreSQL, ? on SQLite), as the db raw methods did.
Construction and identity¶
db |
dbx / gohan |
|---|---|
db.NewRepository(ctx, client, table) |
q, err := dbx.FromClient(client) once, then dbx.NewRepository[T](q, table) |
clickhouse.NewRepository(ctx, conn, table) / chClient.NewRepository(ctx, table) |
dbx.NewRepository[T](chClient.Querier(), table) |
repo.Name() |
repo.Table() |
repo.Db() (*sqlx.DB) |
no equivalent; keep the *db.SqlClient and use client.Db() |
repo.Sql(), repo.SqlDialect(), repo.SqlBuilder() |
q.Dialect() (a gohan.Dialect); gohan builders render with st.Build(q.Dialect()) |
repo.SqlSelect() |
repo.Select() (SELECT <T's columns> FROM <table>) |
repo.SqlInsert() / SqlUpdate() / SqlDelete() |
gohan.Insert(repo.Table()) / gohan.Update(repo.Table()) / gohan.Delete(repo.Table()), run with repo.Exec(ctx, st) |
repo.SqlUpdateX(rec) (db/qb builder) |
repo.Update(ctx, rec, where, opts...); qb options map to gohan.IncludeFields, gohan.ExcludeFields, gohan.SkipZeroValues() (IncludeZeroValues(false)) and gohan.WithAutoFields() (UpdateAutoFields(true)) |
repo.Do(ds, target...) |
by statement kind: repo.Get/repo.List/dbx.Query[D] for a SELECT, repo.Exec(ctx, st) for a write |
db.RegisterFactory, db.RegisterDialect |
no equivalent; a new driver name is mapped with gohan.Register(driverName, dialect) |
Reads and counts¶
db |
dbx / gohan |
|---|---|
repo.FetchOne(ds, &rec) |
rec, err := repo.Get(ctx, repo.Select().Where(...)) (adds LIMIT 1, like FetchOne) |
repo.Fetch(ds, &list) |
list, err := repo.List(ctx, sel); repo.List(ctx, nil) for every row |
repo.FetchRecord(db.FV{...}, &rec) |
repo.GetBy(ctx, map[string]any{...}) |
repo.FetchByKey("id", id, &rec) |
repo.GetBy(ctx, map[string]any{"id": id}) |
repo.FetchWhere(db.FV{...}, &list) |
repo.ListBy(ctx, map[string]any{...}) |
repo.Exists("email", v) |
repo.Exists(ctx, gohan.Col("email").Eq(v)) |
repo.Exists("email", v, "id", id) |
repo.Exists(ctx, gohan.And(gohan.Col("email").Eq(v), gohan.Col("id").Neq(id))) |
repo.Count() |
repo.Count(ctx, nil) |
repo.CountWhere(db.FV{...}) |
repo.Count(ctx, gohan.Match(map[string]any{...})) |
repo.Select(sql, &list, args...) |
q.Select(ctx, &list, sql, args...), or build the query with gohan and use dbx.Query[D](ctx, q, sel) |
a join/aggregate scanned into a DTO with Fetch/Select |
dbx.Query[D](ctx, q, sel) / dbx.QueryOne[D](ctx, q, sel) |
GetBy, ListBy and gohan.Match differ from the db.FV methods in two ways. An empty map
fails with gohan.ErrEmptyMatch (FetchWhere/CountWhere with an empty map matched every row),
and a slice value is compared with =, not expanded to IN (goqu's Eq expanded it): use
gohan.Col(k).In(values...) for lists.
Writes¶
db |
dbx / gohan |
|---|---|
repo.Insert(rec) / repo.Insert([]*T{...}) |
repo.Insert(ctx, rec) / repo.Insert(ctx, recs...) |
records that omit different omitnil/omitempty columns |
repo.WithGroupedInserts().Insert(ctx, recs...) (PostgreSQL, SQLite) |
repo.InsertReturning(rec, fields, &out) |
out, err := repo.InsertReturning(ctx, rec) returns a new *T with every column; rec is not modified |
repo.InsertReturning(rec, []string{"id"}, &id) |
sqlStr, args, err := gohan.Insert(repo.Table()).Rows(rec).Returning("id").Build(q.Dialect()), then id, err := q.QueryInt64(ctx, sqlStr, args...) |
SqlInsert().OnConflict(goqu.DoNothing()) |
inserted, err := repo.InsertIgnore(ctx, rec, conflictCols...) |
SqlInsert().OnConflict(goqu.DoUpdate(...)) |
repo.Upsert(ctx, rec, conflict, update...) / repo.UpsertReturning(ctx, rec, conflict, update...) |
ClickHouse repo.InsertAsync(rec) |
no equivalent; call chClient.Conn.AsyncInsert directly |
Updates¶
db |
dbx / gohan |
|---|---|
repo.UpdateRecord(rec, db.FV{"id": id}) |
repo.Update(ctx, rec, gohan.Match(map[string]any{"id": id})) |
repo.UpdateByKey(rec, "id", id) |
repo.Update(ctx, rec, gohan.Col("id").Eq(id)) |
repo.UpdateFields(&T{}, fields, db.FV{...}) |
repo.UpdateFields(ctx, fields, gohan.Match(map[string]any{...})) |
repo.UpdateFieldsReturning(&T{}, fields, where, returnFields, &out) |
rows, err := repo.UpdateReturning(ctx, fields, where): every updated row, all columns |
repo.UpdateReturning(rec, where, returnFields, &out) |
no record-based equivalent; build gohan.Update(repo.Table()).SetRecord(rec).Where(where).Returning(cols...) and scan it with q.Get(ctx, &out, sqlStr, args...), or pass a map to repo.UpdateReturning |
repo.Update(repo.SqlUpdate().Set(goqu.Record{...}).Where(...)) |
repo.UpdateFields(ctx, map[string]any{...}, where) or repo.Exec(ctx, gohan.Update(repo.Table()).Set(col, v).Where(where)) |
| a PATCH map built by hand from a request body | dbx.NewChangeset(repo) + dbx.SetOptional (or cs.Set), then repo.UpdateFields(ctx, cs.Changes(), where) |
| load, modify, save the whole record | changes, err := dbx.Changes(old, updated), then repo.UpdateFields(ctx, changes, where) when len(changes) > 0 |
Update and UpdateFields return the number of rows affected, and UpdateReturning the rows;
the db methods returned only an error.
Deletes¶
db |
dbx / gohan |
|---|---|
repo.DeleteWhere(db.FV{...}) |
repo.Delete(ctx, gohan.Match(map[string]any{...})) |
repo.DeleteByKey("id", id) |
repo.Delete(ctx, gohan.Col("id").Eq(id)) |
repo.Delete(repo.SqlDelete().Where(...)) |
repo.Delete(ctx, where) |
a delete with no WHERE |
repo.Exec(ctx, gohan.Delete(repo.Table()).All()) |
Raw statements¶
db |
dbx / gohan |
|---|---|
repo.RawExec(sql, args...) |
n, err := q.Exec(ctx, sql, args...) |
repo.Exec(ds) |
repo.Exec(ctx, st) for any gohan statement (returns rows affected) |
db.Fetch, db.RawExec and the other package-level db functions |
the matching Repository method, or q.Get/q.Select/q.Exec/q.QueryInt64 with raw SQL |
Transactions¶
db |
dbx / gohan |
|---|---|
tx, _ := repo.NewTransaction(opts) … tx.Commit() / tx.Rollback() |
dbx.WithTx(ctx, q, opts, func(tx dbx.Querier) error { ... repo.With(tx) ... }) |
| the same, when the transaction must stay open across calls | tx, err := q.BeginTx(ctx, opts) (a dbx.TxQuerier), repo.With(tx), then tx.Commit()/tx.Rollback() |
tx.Insert(...), tx.FetchOne(...), ... |
the same dbx methods on repo.With(tx), for any repository |
tx.Db() (*sqlx.Tx) |
no equivalent; run raw SQL with tx.Exec/tx.Get/tx.Select |
| a loop retrying on serialization failures | dbx.WithTxRetry(ctx, q, opts, attempts, fn) |
set_config/SET LOCAL ROLE at the start of a transaction |
pgsql.WithTxSession(ctx, q, settings, role, fn) |
Grid and errors¶
db |
dbx / gohan |
|---|---|
repo.Grid(&T{}) / db.NewGrid(table, &T{}) |
g, err := dbx.NewGrid[T](), built once and reused |
repo.QueryGrid(&T{}, gq, &list) |
list, err := repo.QueryGrid(ctx, g, gq) |
a second hand-built COUNT for the grid total |
list, total, err := repo.QueryGridWithCount(ctx, g, gq); g.Conds(gq) for other aggregates |
grid.Build(ds, gq) |
g.Build(repo.Select(), gq) |
db.EmptyResult(err) |
errors.Is(err, dbx.ErrNotFound) |
err.(db.GridError) |
errors.As(err, &gridErr) with var gridErr dbx.GridError |
db.FV{...} |
map[string]any{...} |
db.Grid → dbx.Grid¶
db.Grid |
dbx.Grid[T] |
|
|---|---|---|
| Construction | db.NewGrid(table, &T{}), or a new grid per repo.QueryGrid call |
dbx.NewGrid[T]() once; AddFilterFunc, WithMaxLimit, WithTiebreaker and WithCaseInsensitiveSearch mutate it, so configure it before concurrent use |
| Addressable fields | any field with a db tag; a field without the right grid flag answers "field is not filterable"/"not sortable" |
only fields with a grid flag; any other field answers "field is not valid". "not filterable"/"not sortable" remain for a field flagged for another grid operation |
| Aliases | a duplicate alias silently overwrites the earlier one | NewGrid fails on a duplicate, empty or "-" alias among grid fields, and on a searchable field that is neither string-kind nor a driver.Valuer |
| Base query | Build(nil, …) selects * from the table |
a nil base fails ("base query is required"); a UNION base fails (wrap it: gohan.From(u.As("u"))); QueryGrid uses repo.Select() |
Nil GridQuery |
panics | GridError "query is required" |
| Sort | SortFields map, applied in Go map order (unstable) |
Sort list ([{"field": ..., "order": ...}]) in the order given, or SortFields in alias order; not both; a repeated field is rejected |
| Default direction | descending | descending (unchanged) |
| Tiebreaker | none | WithTiebreaker(cols...) appends a unique key, ascending, so equal sort values keep a stable order across pages |
| Row cap | Limit == 0 returns every row |
a new grid caps Limit at dbx.DefaultMaxLimit (1000): Limit == 0 or a larger Limit returns at most 1000 rows. WithMaxLimit(n) changes the cap; WithMaxLimit(0) removes it |
| Offset/limit range | unchecked | above math.MaxInt64 is a GridError; GridQuery.Page saturates instead of overflowing |
| Search wildcards | % and _ in the search text are wildcards |
% and _ match literally |
| Case-insensitive search | not available (LIKE only) |
WithCaseInsensitiveSearch(): ILIKE on PostgreSQL and ClickHouse, LIKE on SQLite (whose LIKE already ignores ASCII case) |
| Search on a type without searchable fields | no search condition; unfiltered rows are returned | GridError "no searchable fields" |
| Search text | unbounded; search type only checked when text is present | at most dbx.MaxSearchText (256) bytes; SearchType always range-checked |
| Filter values | anything the JSON decoder produced, passed to goqu | JSON scalars (nil, bool, float64, string, json.Number) or a flat list of at most dbx.MaxFilterValues (1000) of them; anything else is "value is not valid" |
| Filter func results | used as returned | a gohan expression or a map is "value is not valid"; a typed slice becomes an IN list (at most 1000 values); arrays, byte slices and driver.Valuers bind as one value |
| Filter order | Go map order | sorted by alias, so the same query always builds the same SQL |
| Total count | build a second query yourself | QueryGridWithCount / Conds |
| JSON field names | SearchType (the field had a db tag, not a json tag), searchText, filterFields, sortFields, offset, limit |
searchType, plus the new sort; the others are unchanged. Decoding is case-insensitive, so an old client sending "SearchType" still decodes |
GridError |
db.GridError{Scope, Field, Message} |
dbx.GridError, same fields, JSON and Error() text |
| Configuration errors | none | WithMaxLimit above math.MaxInt64 or WithTiebreaker with a column T does not map: a plain error (not a GridError) from ValidQuery and Build |
dbx.GridError scopes and messages:
| Scope | Messages |
|---|---|
query |
"query is required", "base query is required", "base query must not be a UNION; wrap it with gohan.From(q.As(...))", "offset is out of range", "limit is out of range"; QueryGridKeyset only: "offset is not allowed with cursor pagination" |
filter |
"field is not valid", "field is not filterable", "value is not valid" |
sort |
"field is not valid", "field is not sortable", "sort order is not valid", "field is repeated", "use either sort or sortFields, not both"; QueryGridKeyset only: "field cannot be used with cursor pagination", "too many sort fields for cursor pagination" |
search |
"search text too long", "search not allowed", "invalid search type", "no searchable fields" |
cursor |
"cursor is not valid" (QueryGridKeyset only; errors.Is(err, dbx.ErrInvalidCursor) holds) |
An error returned by a GridFilterFunc is passed through unchanged, as in db. For large tables
where OFFSET paging gets slow, see
Cursor pagination in the dbx guide.
goqu → gohan expressions¶
gohan builders are immutable: each call returns a new builder.
| goqu | gohan |
|---|---|
goqu.C("a") |
gohan.Col("a") |
goqu.I("t.a") |
gohan.Col("t.a") (renders "t"."a") |
goqu.T("users"), goqu.T("users").As("u") |
gohan.Table("users"), gohan.Table("users").As("u"); .Col("id") qualifies a column with the table or alias |
goqu.L("x @> ?", v) |
gohan.Raw("x @> ?", v); a literal ? is written ?? (rejected on ClickHouse). Raw SQL text must never come from request data |
goqu.L("1") for a constant |
gohan.Int(1) (inline literal); gohan.Raw("NOW()") for other SQL constants |
goqu.V(v) |
gohan.Val(v) |
goqu.Star() |
gohan.Star() |
goqu.Ex{"a": 1, "b": nil} |
gohan.Match(map[string]any{"a": 1, "b": nil}) (nil renders IS NULL; an empty map fails with gohan.ErrEmptyMatch) |
goqu.Ex{"a": []int{1, 2}} |
gohan.Col("a").In(1, 2): Match compares a slice with =, it does not expand it |
goqu.Ex{"a": goqu.Op{"gt": 5}} |
gohan.Col("a").Gt(5) |
goqu.ExOr{"a": 1, "b": 2} |
gohan.Or(gohan.Col("a").Eq(1), gohan.Col("b").Eq(2)) |
goqu.And(...), goqu.Or(...) |
gohan.And(...), gohan.Or(...); gohan.Not(e) |
.Eq, .Neq, .Gt, .Gte, .Lt, .Lte |
same names; Eq(nil)/Neq(nil) render IS NULL/IS NOT NULL; the argument may be another column (gohan.Col("a").Gt(gohan.Col("b"))) |
.In(...), .NotIn(...) |
.In(values...), .NotIn(values...); a single slice argument is expanded, a *gohan.SelectBuilder becomes a subquery; an empty In renders 1=0 |
.IsNull(), .IsNotNull() |
.IsNull(), .IsNotNull() |
.Like(p), .NotLike(p) |
.Like(p), .NotLike(p); to match user text use .Contains(s), .HasPrefix(s), .HasSuffix(s), which escape % and _ |
.ILike(p), .NotILike(p) |
.ILike(p), .NotILike(p) (gohan.ErrUnsupported on SQLite); .ContainsFold/.HasPrefixFold/.HasSuffixFold work on every dialect |
.Between(goqu.Range(lo, hi)) |
.Between(lo, hi), .NotBetween(lo, hi) |
ds.Order(goqu.C("a").Desc()), ds.OrderAppend(...) |
sel.OrderBy(gohan.Col("a").Desc()) (appends); .Asc(), .NullsFirst(), .NullsLast() |
ds.Limit(n), ds.Offset(n) |
sel.Limit(n), sel.Offset(n) (uint64) |
ds.Select(cols...), ds.SelectAppend(cols...) |
gohan.Select(cols...) or sel.Columns(cols...) (replaces); sel.AddColumns(cols...) (appends) |
ds.Distinct(), ds.GroupBy(...), ds.Having(...) |
sel.Distinct(), sel.GroupBy(...), sel.Having(...) |
ds.Join(goqu.T("p"), goqu.On(goqu.Ex{"p.user_id": goqu.I("u.id")})) |
sel.Join(gohan.Table("p"), gohan.Col("p.user_id").Eq(gohan.Col("u.id"))); also LeftJoin, RightJoin, FullJoin, CrossJoin, JoinUsing, LeftJoinUsing |
subquery as a source: goqu.From(ds.As("s")) |
gohan.From(sel.As("s")) |
subquery as a value, IN (subquery), EXISTS |
gohan.Sub(sel), gohan.Col("a").In(sel), gohan.Exists(sel), gohan.NotExists(sel) |
ds.With("x", sub), ds.Union(other), ds.UnionAll(other) |
sel.With("x", sub) (WithRecursive), sel.Union(other), sel.UnionAll(other) |
goqu.COUNT("*"), goqu.COUNT("id") |
gohan.CountAll(), gohan.Count("id") |
goqu.SUM, goqu.AVG, goqu.MIN, goqu.MAX |
gohan.Sum, gohan.Avg, gohan.Min, gohan.Max |
goqu.Func("lower", goqu.C("a")) |
gohan.Fn("lower", gohan.Col("a")) (the name is trusted input) |
goqu.Cast(goqu.C("a"), "TEXT") |
gohan.Cast(gohan.Col("a"), "TEXT") (the type is trusted input) |
goqu.Case().When(cond, v).Else(w) |
gohan.Case().When(cond, v).Else(w), or .End() without ELSE |
expr.As("n") |
value.As("n") |
ds.ForUpdate(goqu.Wait) |
sel.ForUpdate() (PostgreSQL only; SkipLocked(), NoWait()) |
goqu.Record{"a": 1} in Set/Rows |
gohan.Update(t).Set("a", 1) / .SetMap(m) / .SetRecord(rec); gohan.Insert(t).SetMap(m) / .Rows(recs...) / .Columns(...).Values(...) |
OnConflict(goqu.DoNothing()) |
.OnConflict(cols...).DoNothing() |
OnConflict(goqu.DoUpdate("k", goqu.Record{"b": goqu.I("excluded.b")})) |
.OnConflict("k").DoUpdate(map[string]any{"b": gohan.Excluded("b")}) or .DoUpdateExcluded("b") |
.Returning(...) |
.Returning(cols...) on insert, update and delete |
ClickHouse FINAL, PREWHERE, SAMPLE, ARRAY JOIN, SETTINGS via goqu.L |
sel.Final(), sel.Prewhere(...), sel.Sample(r)/SampleRows(n), sel.ArrayJoin(...)/LeftArrayJoin(...), sel.Settings(m) |
Behaviour changes¶
These are intentional. Code that relies on the old behaviour needs to change, not just its call syntax.
- Unfiltered DELETE/UPDATE refused.
Delete,Update,UpdateFieldsandUpdateReturningfail withgohan.ErrNoWhere, before touching the database, for a nilwhere, and gohan rejects awherethat cannot restrict rows (gohan.And(), an emptyNotIn,gohan.Raw("1=1")). To affect every row, say so:repo.Exec(ctx, gohan.Delete(table).All()).db'sDeleteWhereand record updates also refuse an empty condition, but a goquSqlDelete()/SqlUpdate()dataset withoutWherestill affects every row. - Empty maps are an error.
GetBy,ListByandgohan.Matchfail withgohan.ErrEmptyMatchon an empty map;FetchRecord,FetchWhereandCountWherewith an empty, non-nil map matched every row. Userepo.List(ctx, nil)/repo.Count(ctx, nil)to mean "all rows". - Slices in equality maps are not expanded.
db'sFVmethods used goqu'sEq, which turned a slice intoIN (...);GetBy/ListBy/gohan.Match/Col(...).Eqbind the slice as one value. Usegohan.Col(k).In(values...). ctxon every call, not a context stored at construction: a repository is no longer tied to one context or deadline for its lifetime.- Reads return new values.
Get,InsertReturning,UpsertReturningand the other reads return a new*T; the record you pass is never written to. - Inserts are chunked and atomic.
Insertsplits records at the dialect's bound-argument limit (65535 on PostgreSQL, 32766 on SQLite) and runs several chunks in one transaction, so a multi-chunk insert over a Querier that cannot begin a transaction fails withdbx.ErrTxUnsupported.dbsent one statement, which failed past the limit. Inserting zero records is a no-op;dbreturned an error. - Rows affected are returned.
Update,UpdateFields,DeleteandExecreturn the count thedbmethods discarded. On ClickHouse it is always0, since ClickHouse does not report it: do not branch onn == 0there. dbxselects an explicit column list, neverSELECT *. A struct field with no matching column fails the query ("missing destination name"), asdb'sSELECT *did. A table column the struct does not map is now simply not selected, whereSELECT *fetched it and failed to scan it.- Record shapes that could bind the wrong data are rejected. An embedded pointer struct, an
unexported or
db/ch-tagged embedded struct, an ambiguous promoted field name, or two fields mapped to one column fail atNewRepository(gohan.ErrRecordShape,gohan.ErrDuplicateColumn) instead of building a statement that writes the wrong field. - Tag parts are not trimmed.
db:"id, auto"andgrid:"sort, filter"lose their second option underdbx(see step 2). - ClickHouse records must map the same columns in
dbandchtags, orNewRepositoryfails withclickhouse.ErrRecordMapping. - Grid changes. The 1000-row default cap, literal search wildcards, stricter validation and "field is not valid" for fields without a grid flag are listed in db.Grid → dbx.Grid.
Int(n)for integer constants. An integer bound as a value inside aCASEfed toSUMis sent as text on PostgreSQL and fails withfunction sum(text) does not exist; write the constant withgohan.Int(n), which renders it inline.- PostgreSQL: a bound value alone in a select list may need
gohan.Cast(v, "type"), when PostgreSQL fails with "could not determine data type of parameter". - SQLite identifiers are quoted with backticks, not double quotes: SQLite silently reads an
unknown double-quoted identifier as a string literal, and backticks avoid that. SQLite's
LIKEignores ASCII case; PostgreSQL's and ClickHouse's do not (useILIKEor theFoldhelpers). - Numeric grid filters arrive as
float64. AGridQuerydecoded from JSON carries{"id": 3.9}as3.9, which matchesid = 3on PostgreSQL (pgx truncates). Register aGridFilterFuncon integer columns to reject non-integer input if that matters. - ClickHouse restrictions. A literal
?is rejected in identifiers and inRaw's??form. Values ClickHouse's driver cannot bind safely are rejected withgohan.ErrUnsafeValue: maps, structs other thantime.Time(unless the value is a top-leveldriver.Valueror afmt.Stringer), and error orfmt.Formattervalues, also inside a slice. AUNIONwith an outerORDER BY/LIMIT/OFFSETis wrapped asSELECT * FROM (<union>) ..., andDelete(t).All()renders... WHERE 1. LightweightDELETEdoes not work on Distributed tables or tables with projections.UPDATE,RETURNING,ON CONFLICTand transactions are not available (gohan.ErrUnsupported,dbx.ErrTxUnsupported). - ClickHouse times keep their precision. Statements built through
client.Querier()use named placeholders, and a boundtime.Time(also*time.Timeor adriver.Valuerproducing one) is sent as aDateTime64literal at the smallest exact scale, never below milliseconds, so comparisons are by exact instant. A time in an IANA zone keeps its zone;time.Localandtime.FixedZonetimes are sent as the same instant in UTC. SQL you write yourself with positional?placeholders still gets clickhouse-go's whole-second binding. See ClickHouse in the dbx guide.
Migrating incrementally¶
- One client, both packages.
dbx.FromClienttakes the same*db.SqlClientthatdb.NewRepositoryuses, andclient.Querier()wraps the same ClickHouse connection as the legacy repository. Move one repository and its callers at a time, test, then move the next. - Transactions do not cross packages. A
db.Transactionand adbx.WithTxtransaction are separate transactions on separate connections, and neither can join the other. Move every statement of a transaction todbxtogether. On SQLite withMaxOpenConns == 1, mixing them deadlocks. - Migrations are unchanged.
db/migrationsand the provider migration managers (pgsql.NewMigrationManager,sqlite.NewMigrationManager,clickhouse.NewMigrationManager) take the same*db.SqlClientor ClickHouse*Clientand keep using the legacy repositories internally for their bookkeeping table.dbxhas no migration API: run the migrations as before, then build thedbxrepositories. - Struct types can be shared while both packages are in use, as long as their tags follow
step 2:
db.Repositoryreads the same tag forms.
Worked example¶
A PostgreSQL user repository with CRUD, a PATCH, a transaction writing two tables, and a grid endpoint that returns a page and its total.
Before: db.Repository¶
package users
import (
"context"
"encoding/json"
"net/http"
"strconv"
"time"
"github.com/doug-martin/goqu/v9"
"github.com/oddbit-project/blueprint/db"
)
type User struct {
ID int64 `db:"id" json:"id" goqu:"skipinsert" grid:"sort,filter"`
Email string `db:"email" json:"email" grid:"sort,search,filter"`
Name string `db:"name" json:"name" grid:"sort,search"`
Phone *string `db:"phone" json:"phone"`
Active bool `db:"active" json:"active" grid:"filter"`
CreatedAt time.Time `db:"created_at" json:"createdAt" goqu:"skipinsert" grid:"sort"`
}
type AuditEntry struct {
ID int64 `db:"id" goqu:"skipinsert"`
UserID int64 `db:"user_id"`
Action string `db:"action"`
}
type Repository struct {
ctx context.Context
users db.Repository
}
func NewRepository(ctx context.Context, client *db.SqlClient) *Repository {
return &Repository{ctx: ctx, users: db.NewRepository(ctx, client, "users")}
}
func (r *Repository) ByID(id int64) (*User, error) {
u := &User{}
if err := r.users.FetchByKey("id", id, u); err != nil {
return nil, err
}
return u, nil
}
// Create inserts u and copies the generated id and created_at back into it.
func (r *Repository) Create(u *User) error {
return r.users.InsertReturning(u, []string{"id", "created_at"}, u)
}
// UserPatch is a PATCH body: a nil field is left unchanged.
type UserPatch struct {
Name *string `json:"name"`
Phone *string `json:"phone"`
}
func (r *Repository) Patch(id int64, p UserPatch) error {
fields := db.FV{}
if p.Name != nil {
fields["name"] = *p.Name
}
if p.Phone != nil {
fields["phone"] = *p.Phone // no way to tell "absent" from "null"
}
if len(fields) == 0 {
return nil
}
return r.users.UpdateFields(&User{}, fields, db.FV{"id": id})
}
func (r *Repository) Delete(id int64) error {
return r.users.DeleteByKey("id", id)
}
func (r *Repository) EmailTaken(email string, exceptID int64) (bool, error) {
return r.users.Exists("email", email, "id", exceptID)
}
// Deactivate marks the user inactive and writes an audit entry, atomically.
func (r *Repository) Deactivate(id int64) error {
tx, err := r.users.NewTransaction(nil)
if err != nil {
return err
}
defer func() { _ = tx.Rollback() }()
if err := tx.UpdateFields(&User{}, db.FV{"active": false}, db.FV{"id": id}); err != nil {
return err
}
// tx is bound to "users"; another table needs a hand-built goqu dataset
audit := goqu.Dialect("pgx").Insert("user_audit").
Rows(&AuditEntry{UserID: id, Action: "deactivate"})
if err := tx.Do(audit); err != nil {
return err
}
return tx.Commit()
}
// Page returns one grid page and the number of rows its filters match.
func (r *Repository) Page(q *db.GridQuery) ([]*User, int64, error) {
var rows []*User
if err := r.users.QueryGrid(&User{}, q, &rows); err != nil {
return nil, 0, err
}
// the total needs the same filters and search, without sort and paging
grid, err := r.users.Grid(&User{})
if err != nil {
return nil, 0, err
}
countQ := *q
countQ.SortFields, countQ.Limit, countQ.Offset = nil, 0, 0
sel, err := grid.Build(r.users.SqlSelect().Select(goqu.COUNT(goqu.Star())), &countQ)
if err != nil {
return nil, 0, err
}
total, err := db.Count(r.ctx, r.users.Db(), sel)
if err != nil {
return nil, 0, err
}
return rows, total, nil
}
func GetHandler(repo *Repository) http.HandlerFunc {
return func(w http.ResponseWriter, req *http.Request) {
id, err := strconv.ParseInt(req.PathValue("id"), 10, 64)
if err != nil {
http.Error(w, "invalid id", http.StatusBadRequest)
return
}
u, err := repo.ByID(id)
if db.EmptyResult(err) {
http.NotFound(w, req)
return
}
if err != nil {
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
writeJSON(w, http.StatusOK, u)
}
}
func GridHandler(repo *Repository) http.HandlerFunc {
return func(w http.ResponseWriter, req *http.Request) {
var q db.GridQuery
if err := json.NewDecoder(req.Body).Decode(&q); err != nil {
http.Error(w, "invalid request", http.StatusBadRequest)
return
}
rows, total, err := repo.Page(&q)
if gridErr, ok := err.(db.GridError); ok {
writeJSON(w, http.StatusBadRequest, gridErr)
return
}
if err != nil {
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
writeJSON(w, http.StatusOK, map[string]any{"rows": rows, "total": total})
}
}
func writeJSON(w http.ResponseWriter, status int, v any) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(status)
_ = json.NewEncoder(w).Encode(v)
}
After: dbx¶
package users
import (
"context"
"encoding/json"
"errors"
"net/http"
"strconv"
"time"
"github.com/oddbit-project/blueprint/db"
"github.com/oddbit-project/blueprint/dbx"
"github.com/oddbit-project/blueprint/types/optional"
"github.com/oddbit-project/gohan"
)
// Tags are unchanged: goqu:"skipinsert" still marks a column auto.
type User struct {
ID int64 `db:"id" json:"id" goqu:"skipinsert" grid:"sort,filter"`
Email string `db:"email" json:"email" grid:"sort,search,filter"`
Name string `db:"name" json:"name" grid:"sort,search"`
Phone *string `db:"phone" json:"phone"`
Active bool `db:"active" json:"active" grid:"filter"`
CreatedAt time.Time `db:"created_at" json:"createdAt" goqu:"skipinsert" grid:"sort"`
}
type AuditEntry struct {
ID int64 `db:"id" goqu:"skipinsert"`
UserID int64 `db:"user_id"`
Action string `db:"action"`
}
type Repository struct {
q dbx.Querier
users *dbx.Repository[User]
audit *dbx.Repository[AuditEntry]
grid *dbx.Grid[User]
}
func NewRepository(client *db.SqlClient) (*Repository, error) {
q, err := dbx.FromClient(client)
if err != nil {
return nil, err
}
users, err := dbx.NewRepository[User](q, "users")
if err != nil {
return nil, err
}
audit, err := dbx.NewRepository[AuditEntry](q, "user_audit")
if err != nil {
return nil, err
}
grid, err := dbx.NewGrid[User]()
if err != nil {
return nil, err
}
// configure once, before concurrent use: these mutate the grid
grid.WithMaxLimit(100).WithTiebreaker("id")
return &Repository{q: q, users: users, audit: audit, grid: grid}, nil
}
func (r *Repository) ByID(ctx context.Context, id int64) (*User, error) {
return r.users.GetBy(ctx, map[string]any{"id": id})
}
// Create inserts u and returns the stored row (generated id and created_at
// included); u itself is not modified.
func (r *Repository) Create(ctx context.Context, u *User) (*User, error) {
return r.users.InsertReturning(ctx, u)
}
// UserPatch is a PATCH body: absent, null and a value stay distinct.
type UserPatch struct {
Name optional.Optional[string] `json:"name,omitzero"`
Phone optional.Optional[string] `json:"phone,omitzero"`
}
func (r *Repository) Patch(ctx context.Context, id int64, p UserPatch) error {
cs := dbx.NewChangeset(r.users)
if err := errors.Join(
dbx.SetOptional(cs, "name", p.Name),
dbx.SetOptional(cs, "phone", p.Phone), // Null clears the column
); err != nil {
return err
}
changes := cs.Changes()
if len(changes) == 0 {
return nil
}
_, err := r.users.UpdateFields(ctx, changes, gohan.Col("id").Eq(id))
return err
}
func (r *Repository) Delete(ctx context.Context, id int64) error {
_, err := r.users.Delete(ctx, gohan.Col("id").Eq(id))
return err
}
func (r *Repository) EmailTaken(ctx context.Context, email string, exceptID int64) (bool, error) {
return r.users.Exists(ctx, gohan.And(gohan.Col("email").Eq(email), gohan.Col("id").Neq(exceptID)))
}
// Deactivate marks the user inactive and writes an audit entry, atomically.
func (r *Repository) Deactivate(ctx context.Context, id int64) error {
return dbx.WithTx(ctx, r.q, nil, func(tx dbx.Querier) error {
n, err := r.users.With(tx).UpdateFields(ctx, map[string]any{"active": false}, gohan.Col("id").Eq(id))
if err != nil {
return err
}
if n == 0 {
return dbx.ErrNotFound
}
return r.audit.With(tx).Insert(ctx, &AuditEntry{UserID: id, Action: "deactivate"})
})
}
// Page returns one grid page and the number of rows its filters match.
func (r *Repository) Page(ctx context.Context, q *dbx.GridQuery) ([]*User, int64, error) {
return r.users.QueryGridWithCount(ctx, r.grid, q)
}
func GetHandler(repo *Repository) http.HandlerFunc {
return func(w http.ResponseWriter, req *http.Request) {
id, err := strconv.ParseInt(req.PathValue("id"), 10, 64)
if err != nil {
http.Error(w, "invalid id", http.StatusBadRequest)
return
}
u, err := repo.ByID(req.Context(), id)
if errors.Is(err, dbx.ErrNotFound) {
http.NotFound(w, req)
return
}
if err != nil {
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
writeJSON(w, http.StatusOK, u)
}
}
func GridHandler(repo *Repository) http.HandlerFunc {
return func(w http.ResponseWriter, req *http.Request) {
var q dbx.GridQuery
if err := json.NewDecoder(req.Body).Decode(&q); err != nil {
http.Error(w, "invalid request", http.StatusBadRequest)
return
}
rows, total, err := repo.Page(req.Context(), &q)
var gridErr dbx.GridError
if errors.As(err, &gridErr) {
writeJSON(w, http.StatusBadRequest, gridErr)
return
}
if err != nil {
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
writeJSON(w, http.StatusOK, map[string]any{"rows": rows, "total": total})
}
}
func writeJSON(w http.ResponseWriter, status int, v any) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(status)
_ = json.NewEncoder(w).Encode(v)
}
What changed:
- Every method takes
ctx;NewRepositoryreturns an error, so a bad record type fails at startup. Createreturns the stored row instead of scanning into its argument.Patchcan now clearphone({"phone": null}), and each value is checked againstUserbefore theUPDATEruns.Deactivatewrites both tables through oneWithTx, with no goqu dataset for the second table, and reports a missing user asdbx.ErrNotFound.Pagereturns the total fromQueryGridWithCountinstead of a second hand-built query. The grid caps pages at 100 rows and sorts ties byid.- Handlers match
dbx.ErrNotFoundanddbx.GridErrorwitherrors.Is/errors.As.