Explore the OQGRAPH storage engine in MariaDB Server. Learn how to efficiently manage hierarchical and complex graph data structures, perfect for social networks and bill of materials.
The Open Query GRAPH computation engine, or OQGRAPH as the engine itself is called, allows you to handle hierarchies (tree structures) and complex graphs (nodes having many connections in several directions).
OQGRAPH is unlike other storage engines, consisting of an entirely different architecture to a regular storage engine such as Aria, MyISAM or InnoDB.
It is intended to be used for retrieving hierarchical information, such as those used for graphs, routes or social relationships, in plain SQL.
See Installing OQGRAPH. Note that if the query cache is enabled, OQGRAPH will not use it.
The following documentation is based upon MariaDB 10.0.7 and OQGRAPH v3.
To create an OQGRAPH v3 table, a backing table must first be created. This backing table will store the actual data, and will be used for all INSERTs, UPDATEs and so on. It must be a regular table, not a view. Here's a simple example to start with:
CREATE TABLE oq_backing (
origid INT UNSIGNED NOT NULL,
destid INT UNSIGNED NOT NULL,
PRIMARY KEY (origid, destid),
KEY (destid)
);
Some data can be inserted into the backing table to test with later:
INSERT INTO oq_backing(origid, destid)
VALUES (1,2), (2,3), (3,4), (4,5), (2,6), (5,6);
Now the read-only OQGRAPH table is created. The CREATE statement must match the format below - any difference will result in an error.
CREATE TABLE oq_graph (
latch VARCHAR(32) NULL,
origid BIGINT UNSIGNED NULL,
destid BIGINT UNSIGNED NULL,
weight DOUBLE NULL,
seq BIGINT UNSIGNED NULL,
linkid BIGINT UNSIGNED NULL,
KEY (latch, origid, destid) USING HASH,
KEY (latch, destid, origid) USING HASH
)
ENGINE=OQGRAPH
data_table='oq_backing' origid='origid' destid='destid';
An older format (prior to MariaDB 10.0.7) has the latch field as a SMALLINT rather than a VARCHAR. The format gives an error:
CREATE TABLE oq_old (
latch SMALLINT UNSIGNED NULL,
origid BIGINT UNSIGNED NULL,
destid BIGINT UNSIGNED NULL,
weight DOUBLE NULL,
seq BIGINT UNSIGNED NULL,
linkid BIGINT UNSIGNED NULL,
KEY (latch, origid, destid) USING HASH,
KEY (latch, destid, origid) USING HASH
)
ENGINE=OQGRAPH
data_table='oq_backing' origid='origid' destid='destid';
ERROR 1005 (HY000): Can't create table `test`.`oq_old` (errno: 140 "Wrong create options")
The old, deprecated format can still be used prior to MariaDB 11.5 if the value of the oqgraph_allow_create_integer_latch system variable is changed from its default, FALSE
, to TRUE
.
SET GLOBAL oqgraph_allow_create_integer_latch=1;
CREATE TABLE oq_old (
latch SMALLINT UNSIGNED NULL,
origid BIGINT UNSIGNED NULL,
destid BIGINT UNSIGNED NULL,
weight DOUBLE NULL,
seq BIGINT UNSIGNED NULL,
linkid BIGINT UNSIGNED NULL,
KEY (latch, origid, destid) USING HASH,
KEY (latch, destid, origid) USING HASH
)
ENGINE=OQGRAPH
data_table='oq_backing' origid='origid' destid='destid';
Query OK, 0 rows affected, 1 warning (0.19 sec)
SHOW WARNINGS;
+---------+------+-----------------------------------------------------------------------------------------------------------------------------------+
| Level | Code | Message |
+---------+------+-----------------------------------------------------------------------------------------------------------------------------------+
| Warning | 1287 | 'latch SMALLINT UNSIGNED NULL' is deprecated and will be removed in a future release. Please use 'latch VARCHAR(32) NULL' instead |
+---------+------+-----------------------------------------------------------------------------------------------------------------------------------+
Data is only inserted into the backing table, not the OQGRAPH table.
Now, having created the oq_graph
table linked to a backing table, it is now possible to query the oq_graph
table directly. The weight
field, since it was not specified in this example, defaults to 1
.
SELECT * FROM oq_graph;
+-------+--------+--------+--------+------+--------+
| latch | origid | destid | weight | seq | linkid |
+-------+--------+--------+--------+------+--------+
| NULL | 1 | 2 | 1 | NULL | NULL |
| NULL | 2 | 3 | 1 | NULL | NULL |
| NULL | 2 | 6 | 1 | NULL | NULL |
| NULL | 3 | 4 | 1 | NULL | NULL |
| NULL | 4 | 5 | 1 | NULL | NULL |
| NULL | 5 | 6 | 1 | NULL | NULL |
+-------+--------+--------+--------+------+--------+
The data here represents one-directional starting and ending nodes. So node 2 has paths to node 3 and node 6, while node 6 has no paths to any other node.
There are three fields which can be manipulated: origid
, destid
(the example above uses these two), as well as weight
. To create a backing table with a weight
field as well, the following syntax can be used:
CREATE TABLE oq2_backing (
origid INT UNSIGNED NOT NULL,
destid INT UNSIGNED NOT NULL,
weight DOUBLE NOT NULL,
PRIMARY KEY (origid, destid),
KEY (destid)
);
INSERT INTO oq2_backing(origid, destid, weight)
VALUES (1,2,1), (2,3,1), (3,4,3), (4,5,1), (2,6,10), (5,6,2);
CREATE TABLE oq2_graph (
latch VARCHAR(32) NULL,
origid BIGINT UNSIGNED NULL,
destid BIGINT UNSIGNED NULL,
weight DOUBLE NULL,
seq BIGINT UNSIGNED NULL,
linkid BIGINT UNSIGNED NULL,
KEY (latch, origid, destid) USING HASH,
KEY (latch, destid, origid) USING HASH
)
ENGINE=OQGRAPH
data_table='oq2_backing' origid='origid' destid='destid' weight='weight';
SELECT * FROM oq2_graph;
+-------+--------+--------+--------+------+--------+
| latch | origid | destid | weight | seq | linkid |
+-------+--------+--------+--------+------+--------+
| NULL | 1 | 2 | 1 | NULL | NULL |
| NULL | 2 | 3 | 1 | NULL | NULL |
| NULL | 2 | 6 | 10 | NULL | NULL |
| NULL | 3 | 4 | 3 | NULL | NULL |
| NULL | 4 | 5 | 1 | NULL | NULL |
| NULL | 5 | 6 | 2 | NULL | NULL |
+-------+--------+--------+--------+------+--------+
See OQGRAPH Examples for OQGRAPH usage examples.
This page is licensed: CC BY-SA / Gnu FDL
The Open Query GRAPH computation engine, or OQGRAPH as the engine itself is called, allows you to handle hierarchies (tree structures) and complex graphs (nodes having many connections in several directions).
The OQGRAPH storage engine exists as a separate package in the repositories for MariaDB 10.0.7 and later. On Ubuntu and Debian the package is called mariadb-oqgraph-engine-10.0
or mariadb-plugin-oqgraph
. On Red Hat, CentOS, and Fedora the package is called MariaDB-oqgraph-engine
. To install the plugin, first install the appropriate package and then install the plugin using the INSTALL SONAME or INSTALL PLUGIN commands.
On Debian and Ubuntu, install the package as follows:
sudo apt-get install mariadb-plugin-oqgraph
or (for MariaDB 10.0)
sudo apt-get install mariadb-oqgraph-engine-10.0
Note that OQGRAPH v3 requires libjudy, which is not in the official Red Hat/Fedora repositories. This needs to be installed first, for example:
wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
rpm -Uvh epel-release-6-8.noarch.rpm
Then install the package, as follows:
sudo yum install MariaDB-oqgraph-engine
On either system you can then launch the mysql
command-line client and install the plugin in MariaDB as follows:
INSTALL SONAME 'ha_oqgraph';
More information on this engine is found on the OpenQuery website: graph-engine
This page is licensed: CC BY-SA / Gnu FDL
To compile OQGraph v2 you need to have the boost library with the versions not earlier than 1.40 and not later than 1.55 and gcc version not earlier than 4.5.
OQGraph v3 compiles fine with the newer boost libraries, but it additionally needs the Judy library installed.
When all build prerequisites are met, OQGraph is enabled and compiled automatically. To enable or disable OQGRAPH explicitly, see the generic plugin build instructions.
If OQGRAPH gets compiled properly, there should be a file like:
storage/oqgraph/ha_oqgraph.so
If this is not the case, then you can find out if there is any modules missing that are required by OQGRAPH by doing:
cmake . -LAH | grep -i oqgraph
This page is licensed: CC BY-SA / Gnu FDL
OQGRAPH v3 can be built on Windows. This has been tested using Windows 7, Microsoft Visual Studio Express 2010 (32-bit), Microsoft Windows 64-bit Platform SDK 7.1 (64-bit), the Boost library >= 1.55 and Judy 1.0.5.
Other recent versions of Boost, Judy or MSVC may work but these combinations have not been tested.
Download the source package for Boost 1.55 from the Boost project website, www.boost.org
Download the source package for Judy 1.05 via
Follow the documented instructions for building under Windows from the command line: Building_MariaDB_on_Windows
Ensure that the following variable is set to CMAKE: JUDY_ROOT=path\to\judy\unzipped
See also comments in storage/oqgraph/cmake/FindJudy.cmake
This page is licensed: CC BY-SA / Gnu FDL
CREATE TABLE oq_backing (
origid INT UNSIGNED NOT NULL,
destid INT UNSIGNED NOT NULL,
PRIMARY KEY (origid, destid),
KEY (destid)
);
Some data can be inserted into the backing table to test with later:
INSERT INTO oq_backing(origid, destid)
VALUES (1,2), (2,3), (3,4), (4,5), (2,6), (5,6);
Now the read-only OQGRAPH table is created.
You can use the following syntax:
CREATE TABLE oq_graph
ENGINE=OQGRAPH
data_table='oq_backing' origid='origid' destid='destid';
For the examples on this page, we'll create a second OQGRAPH table and backing table, this time with weight
as well.
CREATE TABLE oq2_backing (
origid INT UNSIGNED NOT NULL,
destid INT UNSIGNED NOT NULL,
weight DOUBLE NOT NULL,
PRIMARY KEY (origid, destid),
KEY (destid)
);
INSERT INTO oq2_backing(origid, destid, weight)
VALUES (1,2,1), (2,3,1), (3,4,3), (4,5,1), (2,6,10), (5,6,2);
CREATE TABLE oq2_graph (
latch VARCHAR(32) NULL,
origid BIGINT UNSIGNED NULL,
destid BIGINT UNSIGNED NULL,
weight DOUBLE NULL,
seq BIGINT UNSIGNED NULL,
linkid BIGINT UNSIGNED NULL,
KEY (latch, origid, destid) USING HASH,
KEY (latch, destid, origid) USING HASH
)
ENGINE=OQGRAPH
data_table='oq2_backing' origid='origid' destid='destid' weight='weight';
A latch
value of 'dijkstras'
and an origid
and destid
is used for finding the shortest path between two nodes, for example:
SELECT * FROM oq_graph WHERE latch='breadth_first' AND origid=1 AND destid=6;
+----------+--------+--------+--------+------+--------+
| latch | origid | destid | weight | seq | linkid |
+----------+--------+--------+--------+------+--------+
| dijkstras| 1 | 6 | NULL | 0 | 1 |
| dijkstras| 1 | 6 | 1 | 1 | 2 |
| dijkstras| 1 | 6 | 1 | 2 | 6 |
+----------+--------+--------+--------+------+--------+
Note that nodes are uni-directional, so there is no path from node 6 to node 1:
SELECT * FROM oq_graph WHERE latch='dijkstras' AND origid=6 AND destid=1;
Empty set (0.00 sec)
Using the GROUP_CONCAT function can produce more readable results, for example:
SELECT GROUP_CONCAT(linkid ORDER BY seq) AS path FROM oq_graph
WHERE latch='dijkstras' AND origid=1 AND destid=6;
+-------+
| path |
+-------+
| 1,2,6 |
+-------+
Using the table oq2_graph
, the shortest path is different:
SELECT GROUP_CONCAT(linkid ORDER BY seq) AS path FROM oq2_graph
WHERE latch='dijkstras' AND origid=1 AND destid=6;
+-------------+
| path |
+-------------+
| 1,2,3,4,5,6 |
+-------------+
The reason is the weight between nodes 2 and 6 is 10
in oq_graph2
, so the shortest path taking into account weight
is now across more nodes.
SELECT GROUP_CONCAT(linkid) AS dests FROM oq_graph WHERE latch='dijkstras' AND origid=2;
+-----------+
| dests |
+-----------+
| 5,4,6,3,2 |
+-----------+
Note that this returns all possible destinations along the path, not just immediate links.
MariaDB supports the leaves
latch value. A latch
value of 'leaves'
and either origid
or destid
is used for finding leaf nodes at the beginning or end of a graph.
INSERT INTO oq_backing(origid, destid)
VALUES (1,2), (2,3), (3,5), (4,5), (5,6), (6,7), (6,8), (2,8);
For example, to find all reachable nodes from origid
that only have incoming edges:
SELECT * FROM oq_graph WHERE latch='leaves' AND origid=2;
+--------+--------+--------+--------+------+--------+
| latch | origid | destid | weight | seq | linkid |
+--------+--------+--------+--------+------+--------+
| leaves | 2 | NULL | 4 | 2 | 7 |
| leaves | 2 | NULL | 1 | 1 | 8 |
+--------+--------+--------+--------+------+--------+
And to find all nodes from which a path can be found to destid
that only have outgoing edges:
SELECT * FROM oq_graph WHERE latch='leaves' AND destid=5;
+--------+--------+--------+--------+------+--------+
| latch | origid | destid | weight | seq | linkid |
+--------+--------+--------+--------+------+--------+
| leaves | NULL | 5 | 3 | 2 | 1 |
| leaves | NULL | 5 | 1 | 1 | 4 |
+--------+--------+--------+--------+------+--------+
Latch
Alternative
additional where clause fields
Graph operation
NULL
(unspecified)
(none)
List original data
(empty string)
0
(none extra)
List all vertices in linkid column
(empty string)
0
origid
List all first hop vertices from origid in linkid column
dijkstras
1
origid, destid
Find shortest path using Dijkstras algorithm between origid and destid, with traversed vertex ids in linkid column
dijkstras
1
origid
Find all vertices reachable from origid, listed in linkid column, and report sum of weights of vertices on path to given vertex in weight
dijkstras
1
destid
Find all vertices from which a path can be found to destid, listed in linkid column, and report sum of weights of vertices on path to given vertex in weight
breadth_first
2
origid
List vertices reachable from origid in linkid column
breadth_first
2
destid
List vertices from which a path can be found to destid in linkid column
breadth_first
2
origid, destid
Find shortest path between origid and destid, report in linkid column
leaves
4
origid
List vertices reachable from origid, that only have incoming edges
leaves
4
destid
List vertices from which a path can be found to destid, that only have outgoing edges
leaves
4
origid, destid
Not supported, will return an empty result
The use of integer latch commands is deprecated and may be phased out in a future release. Currently, numeric values in the strings are interpreted as aliases, and use of an integer column can be optionally allowed, for the latch commands column.
The use of integer latches is controlled using the oqgraph_allow_create_integer_latch system variable.
This page is licensed: CC BY-SA / Gnu FDL