Backend indexes¶
Every upsert looks up the stored vertex with the same identity, and without an
index on the identity fields each lookup scans the whole vertex type. This page
is for anyone who writes to a graph database with GraFlo: it says which indexes
GraFlo creates on each backend when it defines the schema, which ones you
declare yourself, and how Neo4j, Memgraph and FalkorDB match edges. After
reading it you know what to put in the db_profile block and what to leave to
GraFlo.
Identity indexes and secondary indexes¶
GraFlo distinguishes two kinds of vertex index:
- The identity index covers the fields a vertex type upserts on: its
natural key, or
idfor hash, funnel, blank and assigned vertices (see Vertex identity). GraFlo creates it on every database backend. You do not declare it. - Secondary indexes speed up other lookups. You declare them in the
database profile of the manifest,
schema.db_profile: per vertex type undervertex_indexes, per edge underedge_specs[*].indexes.
schema:
db_profile:
vertex_indexes:
machine:
- fields: [plant, model]
edge_specs:
- source: work_order
target: machine
relation: targets
indexes:
- fields: [opened_at]
An index entry takes fields and, optionally, unique (default false),
type (persistent, the default, hash, skiplist or fulltext) and
name. Every backend builds persistent, hash and skiplist as a plain
index. Only ArangoDB builds fulltext; on any other database a fulltext
index is refused when the schema is applied, with the vertex or edge it is
declared on. The file backend builds no index and ignores it. With
unique: true, the index is a uniqueness constraint over its fields together,
and a write that repeats their values is rejected.
An index over several fields is one index over them, in the order you list them; put the field your lookups filter on most first. Some backends cannot build every index: TigerGraph builds no edge index, no index over several fields and no uniqueness constraint, Memgraph builds edge indexes over one field and without uniqueness, and NebulaGraph builds no uniqueness constraint. An index you declare that the backend cannot build is refused when the schema is applied, naming the vertex or edge and the fields. An index GraFlo derives itself, for an identity or a secondary identity, is skipped there with a warning instead.
sparse and deduplicate are ArangoDB index options; the other backends
ignore them.
An edge_specs entry can also set relation_name, the name the database
stores the relation under. TigerGraph native inverses are set per relation in
db_profile.native_inverses, not on a spec; see Directed, undirected, and
bidirectional edges.
Indexes from secondary identities¶
Every secondary identity on a vertex adds
one non-unique index over its fields to db_profile.vertex_indexes when the
schema is loaded. You do not declare it. If vertex_indexes already has an
index on the same fields, GraFlo keeps yours and adds nothing.
The index is more than an optimization. Endpoint lookups filter on those fields, and on NebulaGraph a filtered lookup cannot run without a tag index.
The index is non-unique because a secondary identity is only softly unique: a
unique index would reject exactly the duplicate values that the
ambiguity policy
exists to handle. An index you declare yourself on the same fields keeps the
unique setting you gave it.
TigerGraph accepts indexes on a single field only. A composite secondary identity logs a warning and gets no index there; the lookup does not depend on it, because GraFlo finds TigerGraph endpoints with an interpreted GSQL query.
NebulaGraph refuses a tag index on a string column unless the index gives a
length. GraFlo gives every string column in a tag index a length of 256. That
includes untyped properties and DATETIME and UUID properties, which
NebulaGraph stores as strings. When NebulaGraph rejects an index anyway, GraFlo
logs a warning, because every filtered read on that tag then fails with
IndexNotFound.
Per backend¶
For the same limits at a glance, beside the other features of each backend, see Choosing a backend.
| Backend | Identity index | Declared vertex indexes | Declared edge indexes |
|---|---|---|---|
| Neo4j | A uniqueness constraint over the identity fields, created with the other indexes. If the server refuses a composite constraint, a plain index with a warning | CREATE INDEX, or a uniqueness constraint when unique: true; a constraint the server refuses is an error |
A relationship index, or a uniqueness constraint when unique: true |
| Memgraph (3.2 or later) | One index over the identity fields and a uniqueness constraint over them, created with the other indexes | One index over the fields, plus a uniqueness constraint when unique: true |
CREATE EDGE INDEX ON :<relation>(<field>), one field only, never unique |
| FalkorDB | Each identity field indexed and a uniqueness constraint over them, created with the other indexes | Each field indexed (FalkorDB keeps one index per label), plus a uniqueness constraint when unique: true |
The same, on the relationship type |
| NebulaGraph | A tag index over the identity fields, always created, because LOOKUP and filtered MATCH need it |
A tag index; never unique | An edge index; never unique |
| ArangoDB | A unique persistent index over the identity fields, created with the collection; none when the identity is _key, which ArangoDB indexes itself |
As declared, with unique and type applied |
As declared, on each edge collection |
| TigerGraph | The primary key (PRIMARY_ID, or PRIMARY KEY for a composite identity), which TigerGraph indexes itself |
One field only, never unique | None |
| PostgreSQL | The vertex table's PRIMARY KEY |
CREATE INDEX, or CREATE UNIQUE INDEX when unique: true |
The same, over the edge's property columns. Every edge table also gets an index on its target columns, and a unique index over its endpoint columns and the edge's identity properties, which covers its identities |
| GraFlo file backend | None | None | None |
On ArangoDB, TigerGraph and PostgreSQL the identity is covered when the collection, vertex type or table is created. On Neo4j, Memgraph, FalkorDB and NebulaGraph, GraFlo creates the identity index together with the declared indexes when it defines the schema.
On PostgreSQL an index name longer than 63 bytes, the identifier limit, is shortened to a head and an 8-hex digest of the whole name, so two long names sharing a prefix stay two indexes.
Defining indexes without a schema¶
The low-level define_vertex_indexes call takes the schema as an optional
argument, and only the schema says which fields form each identity. Called
without it, Neo4j, Memgraph and FalkorDB get no identity index and GraFlo logs
a warning; NebulaGraph gets one anyway, and PostgreSQL gets no secondary
indexes. GraphEngine.define_schema and define_and_ingest always pass the
schema. Pass it yourself when you call define_vertex_indexes or
define_indexes directly.
Writing never creates an index, on any backend. Upserting into a database whose schema was not defined is still correct, but each lookup scans.
Edge upserts and MERGE (Neo4j, Memgraph, FalkorDB)¶
An edge is identified by its endpoints and its relation. On the Cypher
backends it is written with MERGE: the endpoints are matched on their vertex
identities, and the relationship on its type. Without identities there is one
edge per pair of endpoints and relation, and the last record's properties
overwrite the stored ones. Properties are written, never matched on.
To keep several edges between the same two vertices, declare which properties
tell them apart: identities: [[source, target, since]] merges on since as
well. GraFlo takes those names from the first entry of identities, leaving
out the source and target tokens and turning a relation token into the
relationship's relation property. A record that lacks one of them matches an
edge that lacks it too.
PostgreSQL keys every edge table on the same properties, less a relation
token, since each relation has its own table: a unique index over the endpoint
columns and the identity properties, NULLS NOT DISTINCT, so it needs
PostgreSQL 15 or later. An edge write that finds its key taken updates the
edge's other properties.
An edge table whose key is missing or covers other columns, as one created by
an earlier version, gets its key at schema definition or on its first ingest.
That fails with a clear error if the table holds rows the key would merge,
such as parallel edges told apart by a property no identity names.
An edge's identities also feed edge indexes, which are defined with the
schema. The MERGE key is chosen separately, when the graph is written. Keep
both in line with the uniqueness you intend for the edge.
What to read next¶
- Vertex identity: what the identity fields are for each identity mode.
- Directed, undirected, and bidirectional edges: how edge direction and inverses are stored on each backend.