All pages
Powered by GitBook
1 of 74

Client Libraries

The MariaDB project offers client libraries to enable users to connect from a given application to MariaDB and MySQL databases

Connect and Query

Database connections are made using a connector (from an application) or a client (interactively or from scripts).

Clients and connectors listed here are supported under MariaDB Corporation Engineering Policies.

Clients and connectors listed here are compatible with:

  • MariaDB database products (including Enterprise Server and MaxScale)

  • MariaDB Community Server

MariaDB Connectors

MariaDB Connectors are available for many popular programming languages.

Programming Language / Interface
Connector

C

MariaDB Connector/C

C++

MariaDB Connector/C++

Java - JDBC

MariaDB Connector/J

Java - R2DBC

MariaDB Connector/R2DBC

JavaScript

MariaDB Connector/Node.js

ODBC

MariaDB Connector/ODBC

Python

MariaDB Connector/Python

MariaDB Client

MariaDB Client can be used interactively or within scripts.

MariaDB Client is included with distributions of MariaDB database products.

Compatible third-party clients exist but are not listed here.

Executable Filename
Purpose

mariadb, mysql

Connect from the command-line

For additional information about MariaDB Client, see MariaDB Client.

Tools and Utilities

Tools and utilities listed here are included with distributions of MariaDB database products and make a client connection.

Command-Line Executable
Purpose

mariadb-admin, mysqladmin

Check configuration and current status

mariadb-backup, mariadb-backup

Create and restore physical backups (including Aria, InnoDB, MyISAM, MyRocks)

mariadb-binlog, mysqlbinlog

Read binary logs or relay logs

mariadb-check, mysqlcheck

Perform table maintenance operations

mariadb-dump, mysqldump

Create logical backups

mariadb-import, mysqlimport

Load table data from CSV, TSV, and other text file formats

mariadb-show, mysqlshow

Display databases, tables, table columns, indexes

mariadb-slap, mysqlslap

Generate client load for testing

Business Intelligence (BI)

MariaDB database products are accessible from business intelligence (BI) platforms, including:

BI Platform
Detail

Microsoft Power BI

MariaDB Direct Query Adapter for Microsoft Power BI enables Microsoft Power BI Desktop users to remotely connect to and query their MariaDB database, including on MariaDB SkySQL, without downloading the entire data set to their local machine.

This page is: Copyright © 2025 MariaDB. All rights reserved.

Proxy Protocol Support

The proxy protocol allows proxy programs to relay the IP of the clients to the server programs. It is important in case of MariaDB, since IP information is actually a part of user identity.

How Proxy Protocol Works

As per the proxy protocol specification, the connecting client can prefix its first packet with a proxy protocol header. The server will parse the header and assume the client's IP address is the one set in the proxy header.

For example, if a client sends the proxy header (V1, text) which is "PROXY TCP4 192.168.0.1 192.168.0.11 56324 443\r\n", the server, after parsing, assumes the client's IP is 192.168.0.1

MariaDB server understands both Version 1 (text) and Version 2 (binary) of the proxy header.

If the protocol specified by the version 1 header is "UNKNOWN", MariaDB server will treat the connection as if proxy protocol was disabled. This can be used when a valid proxy protocol header is needed but there is no client to proxy, for example when a proxy does a health check.

Enabling Proxy Protocol in MariaDB Server

To enable use of the proxy protocol, it is necessary to specify subnetworks that are allowed to send proxy headers, using the proxy-protocol-networks server variable.

proxy-protocol-networks is a either comma-separated list of (sub)networks or IP addresses. One also can use 'localhost' in this list, which means Unix domain socket/named pipe/shared memory connections are allowed as well. Or, proxy-protocol-networks can be set to *, meaning that proxy headers are allowed from any client.

Note that a client running on a host within an allowed proxy network or an IP address can itself pretend as being connected from any IP address whatsoever and thus can possibly impersonate other users. Generally, you should limit shell access to proxy hosts to a minimum. And remember, with proxy-protocol-networks=* every host is a proxy host.

Example in my.ini/my.cnf

proxy-protocol-networks=::1, 192.168.0.0/16, localhost

allows IPv6 connections from local machine ::1, from IP addresses starting with 192.168, and from connections made with Unix domain sockets or named pipes.

Client-Side Support for Proxy Protocol

Since the functionality is suited only to very specific proxy-like programs, most client APIs do not provide support for sending proxy headers. One exception is Connector/C version 3 or later. One can now use mysql_optionsv():

mysql_optionsv(mysql, MARIADB_OPT_PROXY_HEADER, header,  header_size)

prior to mysql_real_connect() or mysql_connect(), to send the header. In the call above _header_ is the proxy header with the type void *, and _header_size_ is its size in bytes (type is size_t).

Example

const char *hdr="PROXY TCP4 192.168.0.1 192.168.0.11 56324 443\r\n";
mysql_optionsv(mysql, MARIADB_OPT_PROXY_HEADER, hdr,  strlen(hdr));

Using Proxy Protocol with MariaDB MaxScale

If you want to use proxy protocol with MaxScale:

  • Add the IP address of the MaxScale server to proxy-protocol-networks

  • In maxscale.cnf, add the proxy_protocol parameter for all configured servers

Once configured, MaxScale will proxy the credentials from the client to the server.

See Also

  • Proxy Protocol

This page is licensed: CC BY-SA / Gnu FDL

Client/Server Protocol

Description of the protocol used between clients and the MariaDB or MySQL server

0 - Packet

Client - server exchanges are done using the following format:

Standard packet

The standard MySQL/MariaDB packet has a 4 bytes header + packet body.

  • int<3> packet length

  • int<1> sequence number

  • byte packet body

Packet length is the length of the packet body. Packet length size cannot be more than 3 bytes length value. The actual length of the packet is calculated as from the 3 bytes as length = byte[0] + (byte[1]<<8) + (byte[2]<<16). The maximum size of a packet (with all 3 bytes 0xff) is 16777215 , or 2^24-1 or 0xffffff, or 16MB-1byte.

The sequence number indicates the exchange number when an exchange demands different exchanges. Whenever the client sends a query, the sequence number is set to 0 initially, and is incremented if clients need to split packets. In more complex situations, when the client and server exchange several packets, e.g authentication handshake, the rule of thumb for clients is to set sequence nr = (last seq.nr from received server packet + 1)

Example: Sending a COM_PING packet COM_PING body has only one byte (0x10):

01 00 00 00 10

The server will then return an OK_Packet response with a sequence number of 1.

Packet splitting

As mentioned, the packet length is 3 bytes making a maximum size of (2^24 -1) bytes or 16Mbytes-1byte. But the protocol allows sending and receiving larger data. For those cases, the client can send many packets for the same data, incrementing the sequence number for each packet.

The principle is to split data by chunks of 16MBytes. When the server receives a packet with 0xffffff length, it will continue to read the next packet. In case of a length of exactly 16MBytes, an empty packet must terminate the sequence.

Example max_allowed_packet is set to a value > to 40 Mbytes Sending a 40M bytes packet body : First packet will be :

ff ff ff 00 ...

second packet will be

ff ff ff 01 ...

third packet will be

02 00 80 02 ...

The client must be aware of the max_allowed_packet variable value. The server will have a buffer to store the body with a maximum size corresponding to this max_allowed_packet value. If the client sends more data than max_allowed_packet size, the socket will be closed.

Note that data of exact size 2^24 -1 byte must be sent in 2 packets, the first one with length prefix 0xffffff, and the second one with length 0 (length byte 0x000000, seqno incremented). Generally, if data length is an exact multiple of 2^24-1, it must always be followed by an empty packet.

Compressed packet

For slow connections, the packet can be compressed. This is activated after the handshake-response-packet when the client indicates [[1-connecting-connecting#capabilities|COMPRESS] capability with the server having this functionality too.

When activated, packets will be composed of 7 bytes a compress header + data. The compression algorithm used is ZLIB, widely available and supported by many languages and runtimes.

  • int<3> compressed packet length

  • int<1> compression protocol sequence number

  • int<3> uncompress body length

  • byte compressed body

    • compressed body contains one or many standard packets but partial packets can also be sent:

      • one or many standard packets :

        • int<3> packet length

        • int<1> sequence number

        • byte packet body

Since compress body can contain many "standard packets", compress sequence number is incremented separately from sequence number. If the length of the uncompressed payload exceeds 0xffffff bytes, the uncompressed payload must be sent in several compressed protocol packets.

For small packets, using compression won't be efficient, so the client can choose to send uncompressed data. That is done by setting the compressed packet length to the real length and the uncompressed packet length to 0. (Data must then be uncompressed).

Example: Sending a COM_PING packet COM_PING body when COMPRESS is enabled. This is 1 byte of data that has then no reason to be compressed, so:

01 00 00 00 00 00 00 01 00 00 00 10

The server will then return an OK_Packet response with a compress sequence number of 1, and a sequence number of 1.

Compression packet splitting

The server will uncompress data and then must have the same packet than if there was no compression. If data size needs splitting, it's better to separate compress packet.

compress_packet

This page is licensed: CC BY-SA / Gnu FDL

MariaDB Protocol Differences with MySQL

Here is a list of the differences between MariaDB and MySQL in terms of protocol, in order to help community driver maintainers.

MariaDB Capabilities Extension

MariaDB/MySQL servers can advertise feature support using capabilities. To expand the capabilities beyond the original 4 bytes, MariaDB utilizes 4 bytes, unused by MySQL, in the Initial handshake packet (server capabilities 3rd part). In order to avoid incompatibility in the future, those 4 bytes have to be read only if capability CLIENT_MYSQL is not SET (server then being MariaDB).

Enhanced Capabilities

  • MARIADB_CLIENT_CACHE_METADATA: Enables clients to cache metadata and avoid repeated network transmissions. (since MariaDB 10.6.0)

  • MARIADB_CLIENT_EXTENDED_METADATA : Provides more detailed column metadata information for specific data types. (since MariaDB 10.5.2)

  • MARIADB_CLIENT_STMT_BULK_OPERATIONS: Introduces a dedicated command, COM_STMT_BULK_EXECUTE, for efficient batch execution of statements. (since MariaDB 10.2.0)

  • MARIADB_CLIENT_BULK_UNIT_RESULTS: Allows for individual result sets for each bulk operation. (since MariaDB 11.5.1)

See Connection Capabilities.

Prepare Statement Skipping Metadata

since MariaDB 10.6.0

Prepared statement metadata, which typically remains unchanged except during table alterations, can be cached by clients when the MARIADB_CLIENT_CACHE_METADATA capability is enabled. The server won't then send them again, unless they change. This significantly improves the performance of subsequent executions, especially for large metadata sets.

When MARIADB_CLIENT_CACHE_METADATA capability is set, the Resultset Column count packet format indicates if metadata follows or is skipped:

  • int column count

  • if (MARIADB_CLIENT_CACHE_METADATA capability set) int<1> metadata follows (0 / 1)

Example

java code:

stmt.execute("CREATE TABLE test_table (id int, val varchar(32))");
stmt.execute("INSERT INTO test_table VALUES (1, 'a'), (2, 'b')");
try (PreparedStatement prep = sharedConnBinary.prepareStatement("SELECT * FROM test_table WHERE id = ?")) {
    prep.setInt(1, 1);
    prep.executeQuery();
}

Results with metadata caching:

Column count packet:
       +--------------------------------------------------+
       |  0  1  2  3  4  5  6  7   8  9  a  b  c  d  e  f |
+------+--------------------------------------------------+------------------+
|000000| 02 00 00 01 02 00                                | ......           |
+------+--------------------------------------------------+------------------+
row:
       +--------------------------------------------------+
       |  0  1  2  3  4  5  6  7   8  9  a  b  c  d  e  f |
+------+--------------------------------------------------+------------------+
|000000| 08 00 00 02 00 00 01 00  00 00 01 61             | ...........a     |
+------+--------------------------------------------------+------------------+

OK_Packet with a 0xFE header:
       +--------------------------------------------------+
       |  0  1  2  3  4  5  6  7   8  9  a  b  c  d  e  f |
+------+--------------------------------------------------+------------------+
|000000| 07 00 00 03 FE 00 00 22  00 00 00                | ......."...      |
+------+--------------------------------------------------+------------------+

same without metadata caching:

Column count packet:
       +--------------------------------------------------+
       |  0  1  2  3  4  5  6  7   8  9  a  b  c  d  e  f |
+------+--------------------------------------------------+------------------+
|000000| 01 00 00 01 02                                   | .....            |
+------+--------------------------------------------------+------------------+

Column Definition packet:
       +--------------------------------------------------+
       |  0  1  2  3  4  5  6  7   8  9  a  b  c  d  e  f |
+------+--------------------------------------------------+------------------+
|000000| 33 00 00 02 03 64 65 66  05 74 65 73 74 6A 0A 74 | 3....def.testj.t |
|000010| 65 73 74 5F 74 61 62 6C  65 0A 74 65 73 74 5F 74 | est_table.test_t |
|000020| 61 62 6C 65 02 69 64 02  69 64 0C 3F 00 0B 00 00 | able.id.id.?.... |
|000030| 00 03 00 00 00 00 00                             | .......          |
+------+--------------------------------------------------+------------------+

Column Definition packet:
       +--------------------------------------------------+
       |  0  1  2  3  4  5  6  7   8  9  a  b  c  d  e  f |
+------+--------------------------------------------------+------------------+
|000000| 35 00 00 03 03 64 65 66  05 74 65 73 74 6A 0A 74 | 5....def.testj.t |
|000010| 65 73 74 5F 74 61 62 6C  65 0A 74 65 73 74 5F 74 | est_table.test_t |
|000020| 61 62 6C 65 03 76 61 6C  03 76 61 6C 0C FF 00 80 | able.val.val.... |
|000030| 00 00 00 FD 00 00 00 00  00                      | .........        |
+------+--------------------------------------------------+------------------+

row:
       +--------------------------------------------------+
       |  0  1  2  3  4  5  6  7   8  9  a  b  c  d  e  f |
+------+--------------------------------------------------+------------------+
|000000| 08 00 00 04 00 00 01 00  00 00 01 61             | ...........a     |
+------+--------------------------------------------------+------------------+

OK_Packet with a 0xFE header:
       +--------------------------------------------------+
       |  0  1  2  3  4  5  6  7   8  9  a  b  c  d  e  f |
+------+--------------------------------------------------+------------------+
|000000| 07 00 00 05 FE 00 00 22  00 00 00                | ......."...      |
+------+--------------------------------------------------+------------------+

Extended Column Information

since MariaDB 10.5.2

When the MARIADB_CLIENT_EXTENDED_METADATA capability is set, column definition packet can include additional type and format information.

  • For geometric fields: Detailed geometric data type (e.g., 'point', 'polygon')

  • For JSON fields: Type 'json'

  • For UUID fields: Type 'uuid'

Bulk

since MariaDB 10.2.0 - MariaDB 11.5.1 for unit results

The MARIADB_CLIENT_STMT_BULK_OPERATIONS capability enables the COM_STMT_BULK_EXECUTE command for efficient batch processing. However, note that only one result (OK or ERROR) is returned per batch, containing the total affected rows and the first auto-generated ID. For individual results, the MARIADB_CLIENT_BULK_UNIT_RESULTS capability can be set. The server will then return a resultset containing for each unitary results (containing auto generated ids and affected rows)

Example

java code:

Statement stmt = connection.createStatement();
stmt.execute("CREATE TABLE test_table (id int, val varchar(32))");

try (PreparedStatement prep = connection.prepareStatement("INSERT INTO test_table VALUES (?, ?)")) {

    prep.setInt(1, 1);
    prep.setString(2, "a");
    prep.addBatch();

    prep.setInt(1, 2);
    prep.setString(2, "b");
    prep.addBatch();

    prep.executeBatch();
}

client send :

MARIADB_CLIENT_STMT_BULK_OPERATIONS:
       +--------------------------------------------------+
       |  0  1  2  3  4  5  6  7   8  9  a  b  c  d  e  f |
+------+--------------------------------------------------+------------------+
|000000| 1B 00 00 00 FA FF FF FF  FF 80 00 03 00 FD 00 00 | ................ |
|000010| 01 00 00 00 00 01 61 00  02 00 00 00 00 01 62    | ......a.......b  |
+------+--------------------------------------------------+------------------+

server response:

OK_Packet:
       +--------------------------------------------------+
       |  0  1  2  3  4  5  6  7   8  9  a  b  c  d  e  f |
+------+--------------------------------------------------+------------------+
|000000| 2E 00 00 01 00 02 00 02  00 00 00 26 52 65 63 6F | ...........&Reco |
|000010| 72 64 73 3A 20 32 20 20  44 75 70 6C 69 63 61 74 | rds: 2  Duplicat |
|000020| 65 73 3A 20 30 20 20 57  61 72 6E 69 6E 67 73 3A | es: 0  Warnings: |
|000030| 20 30                                            |  0               |
+------+--------------------------------------------------+------------------+

Authentication Plugins

MariaDB has specific authentication methods.

  • ED25519 plugin since MariaDB 10.3.0

  • PARSEC plugin since MariaDB 11.6.1

  • GSSAPI plugin since MariaDB 10.1.10

Redirection

since MariaDB 11.3.1 or Maxscale 25.08.0

MariaDB permits connection redirection.

Use Cases

  • Proxy Scenarios: Connection redirection is particularly beneficial when multiple servers share a single proxy.

  • Server Management: This feature can also be used during planned server shutdowns or restarts, allowing for a graceful transition to a new server.

Connectors can support 2 different levels:

  • On Connection Creation only: The redirection information is included in the initial OK_Packet sent by the server to the client. This allows the client to connect directly to the target server immediately.

  • Anytime Redirection: If redirection information becomes available later, the connector can handle it based on the existing transaction state.

    • No Transaction: If no transaction is in progress, the connector can redirect the connection directly.

    • Transaction in Progress: If a transaction is ongoing, the redirection information is stored until the transaction is completed. The transaction state is determined using server status flags like SERVER_STATUS_IN_TRANS in the "OK_Packet," "ERR_Packet," or "EOF_Packet."

Zero-Configuration SSL

Note: This feature is available since MariaDB 11.4.1

A feature that enables TLS certificate validation without requiring client-side certificate configuration.

Limitations

  • required a non empty password

  • Only supports the following authentication methods:

    • mysql_native_password

    • client_ed25519

    • parsec

Operational Mechanism

Server-Side Process

  1. When no SSL certificates are pre-configured, the server automatically generates a temporary self-signed certificate

  2. During connection establishment, the server embeds a special validation hash in the connection's "OK_Packet" information field

Client-Side Process

  1. The client connector must postpone SSL error handling until the connection phase is complete

  2. The client captures and stores the SHA256 fingerprint of the server's certificate

  3. If SSL errors occur, the client can only use specific authentication plugins (mysql_native_password/ed25519/parsec) to prevent potential password exposure

  4. At connection conclusion, the server sends an OK_Packet with a validation hash

  5. The client generates a hash using:

  • The password hash

  • The server's seed

  • Stored certificate fingerprint

  1. The SSL-error connection proceeds only if the client-generated hash matches the server-provided hash

Password Hash Generation Methods

  • mysql_native_password:

    • Hash generation: SHA1(SHA1(password))

  • ed25519:

    • Uses the Ed25519 cryptographic algorithm for hash generation

  • parsec:

    • Hash generation involves combining

      • 'P' character

      • Number of iterations

      • Salt

      • Raw public key

Initial Session Tracking

MySQL 5.7.0 and MariaDB 10.2.2 support session tracking when the CLIENT_SESSION_TRACK capability is set.

One difference is that since MariaDB 11.5.1, connection ending OK_Packet lists all the current variables of tracked variable.

This is usefull for connector having method to set transaction type, retrieving database for example to always have the server current value when changed. This permit to avoid executing some queries when not needed

Example of ending connection OK_Packet :

+--------------------------------------------------+
       |  0  1  2  3  4  5  6  7   8  9  a  b  c  d  e  f |
+------+--------------------------------------------------+------------------+
|000000| A6 00 00 02 00 00 00 02  40 00 00 00 9D 00 0E 0A | ........@....... |
|000010| 61 75 74 6F 63 6F 6D 6D  69 74 02 4F 4E 00 11 09 | autocommit.ON... |
|000020| 74 69 6D 65 5F 7A 6F 6E  65 06 53 59 53 54 45 4D | time_zone.SYSTEM |
|000030| 00 1D 14 63 68 61 72 61  63 74 65 72 5F 73 65 74 | ...character_set |
|000040| 5F 63 6C 69 65 6E 74 07  75 74 66 38 6D 62 34 00 | _client.utf8mb4. |
|000050| 21 18 63 68 61 72 61 63  74 65 72 5F 73 65 74 5F | !.character_set_ |
|000060| 63 6F 6E 6E 65 63 74 69  6F 6E 07 75 74 66 38 6D | connection.utf8m |
|000070| 62 34 00 1E 15 63 68 61  72 61 63 74 65 72 5F 73 | b4...character_s |
|000080| 65 74 5F 72 65 73 75 6C  74 73 07 75 74 66 38 6D | et_results.utf8m |
|000090| 62 34 00 0E 0C 72 65 64  69 72 65 63 74 5F 75 72 | b4...redirect_ur |
|0000a0| 6C 00 01 06 05 74 65 73  74 6A                   | l....testj       |
+------+--------------------------------------------------+------------------+

It indicates:

  • autocommit = ON

  • time_zone = SYSTEM

  • character_set_client = utf8mb4

  • character_set_connection = utf8mb4

  • character_set_results = utf8mb4

  • redirect_url =

A connector knows that character_set_client set to utf8mb4, then could avoid executing a command like "SET NAMES utf8mb4" commands for example.

MySQL Features Not Supported

  • X protocol is not supported

Unsupported features and associate capabilities:

  • CLIENT_OPTIONAL_RESULTSET_METADATA: permits setting no METADATA at all for a connection. See Prepare statement skipping metadata's MariaDB implementation choice.

  • CLIENT_QUERY_ATTRIBUTES adds some metadata attributes

  • CLIENT_ZSTD_COMPRESSION_ALGORITHM permits zstd compression

  • MULTI_FACTOR_AUTHENTICATION Multifactor Authentication capability.

TIPS

Identifying MariaDB Server

MariaDB connectors use specific criteria to determine if a server is a MariaDB instance during the initial handshake process.

The two key indicators used are:

  • Missing CLIENT_MYSQL Capability: MariaDB 10.2 and newer versions do not set the CLIENT_MYSQL capability flag in the initial handshake packet.

  • Server Version String: The server's version string is examined for the presence of the word "mariadb" (ignoring case sensitivity).

The reason is some features like using COM_RESET_CONNECTION has no capability, and depend on the MySQL or MariaDB server version.

Pipelining Prepare Execute

Connectors usually follow a two-step process for prepared statements:

  • Prepare: Send a COM_STMT_PREPARE command to the server, receiving a statement ID in response.

  • Execute: Send a COM_STMT_EXECUTE command, using the statement ID obtained in the previous step.

When the server support MARIADB_CLIENT_STMT_BULK_OPERATIONS capability (since MariaDB 10.2), a specific statement ID value of -1 (or 0xffffffff in hexadecimal) can be used to indicate that the previously prepared statement could be reused. This enables connectors to pipeline the preparation and execution steps into a single request:

  • Send a COM_STMT_PREPARE then a COM_STMT_EXECUTE with statement ID -1 (0xffffffff) commands to the server.

  • Read the prepare and execute responses

If the COM_STMT_PREPARE command returns an error (ERR_Packet), the subsequent COM_STMT_EXECUTE with statement ID -1 will also fail and return an error.

By eliminating the round trip for the separate COM_STMT_EXECUTE command, this approach improves performance for the first execution.

Traditionally, connectors send COM_STMT_PREPARE, wait for results, then execute COM_STMT_EXECUTE with statement_id received from the prepare result.

This description has been done for COM_STMT_EXECUTE, but COM_STMT_BULK_EXECUTE works exactly the same way.

Query Timeout

Since MariaDB 10.1.2

Setting a timeout for all commands can be set using SET max_statement_time=XXX with XXX in seconds.

Setting it for a specific query can be done using SET STATEMENT max_statement_time=XXX FOR ...

Collations

Connectors don't care about collations, but normally want to ensure charset in connection exchanges.

The only good solution is to use SET NAMES utf8mb4 or SET NAMES utf8mb4 COLLATE someUtf8mb4collation

If supporting session tracking, connectors can check if charset of initial tracked variable 'character_set_connection' corresponds to the expected value, then permit skipping this SET NAMES command.

( 'server default collation' from initial handshare packet cannot be trusted, since truncated to one byte. Recent mysql and mariadb collation can go on 2 bytes)

This page is licensed: CC BY-SA / Gnu FDL

Protocol data types

List of possible types

Unknown type:

byte<1>

Fixed-length bytes

byte

Length-encoded bytes

byte

End-of-file length bytes

Integer type:

int<1>

Fixed-length integers

int

Length-encoded integers

String type:

string

Fixed-length strings

string

Null-terminated strings

string

Length-encoded strings

string

End-of-file length strings

Fixed length bytes

The notation is "byte"

A fixed-length bytes stores the value in a series of n bytes.

Length encoded bytes

The notation is "byte" Length encoded bytes are prefixed by a length-encoded integer which describes the length of the byte value, followed by the bytes value.

End of file length bytes

The notation is "byte" Bytes whose length will be calculated by the packet remaining length.

Fixed length integers

Notation is "int"

A fixed-length integer stores the value in a series of n bytes. The least significant byte is always the first byte (little-endian format).

Example

An int<4> with value 2 is stored as02 00 00 00

Length encoded integers

The notation is "int" An integer which depending on its value is represented by n bytes.

The first byte represents the size of the integer:

If the value of first byte is

  • < 0xFB - Integer value is this 1 byte integer

  • 0xFB - NULL value

  • 0xFC - Integer value is encoded in the next 2 bytes (3 bytes total)

  • 0xFD - Integer value is encoded in the next 3 bytes (4 bytes total)

  • 0xFE - Integer value is encoded in the next 8 bytes (9 bytes total)

Fixed-length strings

The notation is "string" Fixed-length strings have a known hardcoded length.

Null-terminated strings

The notation is "string" Null-terminated strings have a variable size and are terminated by a 0x00 character

Length-encoded strings

The notation is "string" Length-encoded strings are prefixed by a length-encoded integer which describes the length of the string, followed by the string value.

Example

An string of 512 "a" will be encoded in 515 bytes :

fc 00 02 97 97 97 97 97 97 97 97 97 97 97 97 97

² .. a a a a a a a a a a a a a

... Null value will be encoded using null (0xfb) length. Empty value will be encoded with a 0 (0x00) length.

End of file length strings

The notation is "string" Strings whose length will be calculated by the packet remaining length. For an example see COM_STMT_PREPARE packet.

This page is licensed: CC BY-SA / Gnu FDL

1 - Connecting

Establishing a connection between client and server

caching_sha2_password Authentication Plugin

Caching SHA256 first send a SHA256 encrypted password. MySQL server has an in-memory cache of SHA256 key for successful authentication. When a cache hit occur, the connection is validated, if not, using some more steps to a process similar to sha256_password.

Caching SHA256 authentication possible exchanges:

  • client send a SHA-2 encrypted password

  • server result in either OK_Packet , ERR_Packet or "fast" authentication result

  • if fast authentication result

    • if connection use SSL (SSLRequest Packet sent)

      • client send a clear password answer

    • else

      • if client doesn't know server RSA public key

        • client sends a public key request

        • server sends a public key response

      • client sends an RSA encrypted password

      • ends with server sending either OK_Packet , ERR_Packet

SHA-2 encrypted password

Encryption is XOR(SHA256(password), SHA256(seed, SHA256(SHA256(password))))

  • byte<32> encrypted password

"fast" authentication result

result of fast authentication.

  • byte authentication result

0x03 value means success authentication. 0x04 value means continue

Client clear password answer

  • string password without encryption

Public key request

Value send is not 0x01 like sha256_password use, but 0x02

  • byte<1> fixed 0x02 value

Public key response

  • byte<1> fixed 0x01 value

  • byte public key data

RSA encrypted password

  • byte<256> RSA encrypted password

RSA encrypted value of XOR(password, seed) using server public key (RSA_PKCS1_OAEP_PADDING).

This page is licensed: CC BY-SA / Gnu FDL

Connecting

Connection is done by many exchanges:

  • (Create socket)

  • If first byte from server is 0xFF:

    • packet is an ERR_Packet, socket has to be closed

  • else

    • Packet is an Initial handshake packet

    • If SSL/TLS connection

      • Client sends SSLRequest packet and switches to SSL mode for sending and receiving the following messages:

    • Client sends Handshake response packet

    • Server sends either:

      • An OK packet in case of success OK_Packet

      • An error packet in case of error ERR_Packet

      • Authentication switch

        • If the client or server doesn't have PLUGIN_AUTH capability:

          • Server sends 0xFE byte

          • Client sends old_password

        • else

          • Server sends Authentication switch request

          • Client may have many exchange with the server according to the Plugin.

        • Authentication switch ends with server sending either OK_Packet or ERR_Packet

Initial Handshake Packet

  • int<1> protocol version

  • string server version (MariaDB server version for 10.X versions is by default prefixed by "5.5.5-". MariaDB 11.0 and later versions do not have a "5.5.5-" default prefix)

  • int<4> connection id

  • string<8> authentication plugin data (1st part)

  • string<1> reserved byte

  • int<2> server capabilities (1st part)

  • int<1> server default collation

  • int<2> status flags

  • int<2> server capabilities (2nd part)

  • if (server_capabilities & PLUGIN_AUTH)

    • int<1> plugin data length

  • else

    • int<1> 0x00

  • string<6> filler

  • if (server_capabilities & CLIENT_MYSQL)

    • string<4> filler

  • else

    • int<4> server capabilities 3rd part . MariaDB specific flags /* MariaDB 10.2 or later */

  • if (server_capabilities & CLIENT_SECURE_CONNECTION)

    • string authentication plugin data 2nd part . Length = max(12, plugin data length - 9)

    • string<1> reserved byte

  • if (server_capabilities & PLUGIN_AUTH)

    • string authentication plugin name

Client Handshake Response

If the client requests a TLS/SSL connection, first response will be an SSL connection request packet, then a handshake response packet. If no TLS is required, client send directly a handshake response packet.

SSLRequest Packet

  • int<4> client capabilities

  • int<4> max packet size

  • int<1> client's default character set and collation

  • string<19> reserved

  • if not (server_capabilities & CLIENT_MYSQL)

    • int<4> extended client capabilities

  • else

    • string<4> reserved

ZERO-CONFIGURATION SSL ENCRYPTION

Automatic Encrypted Connections (MariaDB 11.4+):

Previously, failed SSL connections due to self-signed certificates prevented communication. MariaDB 11.4+ introduces a secondary validation method that works for all servers.

What Happens When SSL Validation Fails?

Even without a valid SSL certificate, the connector can still authenticate by remembering the server's fingerprint (unique identifier). However, it needs to confirm the connection is secure.

Verifying a Secure Connection:

The confirmation method depends on the connection type. When using secure MitM-proof methods, like Unix sockets, connector can automatically validate the connection. Otherwise, a shared secret is used.

Shared Secret for Secure Connection:

The shared secret is only used if the authentication plugin password is hashable (e.g., mysql_native_password , client_ed25519 or parsec) and not empty.

It's calculated by hashing the user's hash password with the authentication seed and the server fingerprint.

Password hash is generated depending on authentication plugin:

  • ed25519 : identical to password encryption

  • native password : identical to password encryption

  • parsec: ext-salt + raw ed25519 public key

Server 11.4+ Confirmation Details:

For servers running MariaDB 11.4 or later, the final confirmation packet contains:

  • int<1> encryption (actually only 0x01 = SHA256 encryption)

  • byte shared secret.

Matching the Shared Secret:

If the calculated shared secret matches the received one, the SSL connection is considered valid (host validation is not needed). Otherwise, the connection must be closed for security reasons.

Handshake Response Packet

  • int<4> client capabilities

  • int<4> max packet size

  • int<1> client's default character set and collation

  • string<19> reserved

  • if not (server_capabilities & CLIENT_MYSQL)

    • int<4> extended client capabilities

  • else

    • string<4> reserved

  • string username

  • if (password)

    • if (server_capabilities & PLUGIN_AUTH_LENENC_CLIENT_DATA)

      • string authentication data

    • else if (server_capabilities & CLIENT_SECURE_CONNECTION)

      • int<1> length of authentication response

      • string authentication response (length is indicated by previous field)

    • else

      • string authentication response null ended

  • else

    • string<1>\0 (empty password)

  • if (server_capabilities & CLIENT_CONNECT_WITH_DB)

    • string default database name

  • if (server_capabilities & CLIENT_PLUGIN_AUTH)

    • string authentication plugin name

  • if (server_capabilities & CLIENT_CONNECT_ATTRS)

    • int size of connection attributes

    • while packet has remaining data

      • string key

      • string value

Server Response to Handshake Response Packet

The server responds with an OK_packet, an ERR_packet or an Authentication Switch Request packet.

Authentication Switch Request

(If client and server support CLIENT_AUTH capability)

  • int<1> 0xFE : Authentication switch request header

  • string authentication plugin name

  • byte authentication plugin data

Plugin List

mysql_old_password Plugin

deprecated send a 8 byte encrypted password

authentication plugin data format :

  • byte<8> 8-byte seed

Client response :

  • string old format encrypted password

mysql_clear_password Plugin

Since password is transmitted in clear, this has be used only when using SSL connection

send clear password to server

Client response :

  • string password without encryption

mysql_native_password Plugin

SHA-1 encrypted password with server seed

authentication plugin data format :

  • string seed

Client response :

  • byte sha1 encrypted password

The password is encrypted with: SHA1( password ) ^ SHA1( seed + SHA1( SHA1( password ) ) )

dialog Plugin (PAM)

Interactive exchanges to permit fill passwords - for example for 2-Step authentication.

authentication plugin data format :

  • byte<1> password type

  • string prompt message

The server can send one or many requests. For each of them, the client must display this prompt message to the user, to permit the user to type requested information, then send it to the server in string format. Password type indicate answer format ( 2 means "read the input with the echo enabled", 4 means "password-like input, echo disabled")

First authentication format (from authentication switch packet) can be empty since 10.4.

This end when server send an EOF_Packet, OK_Packet or ERROR_packet.

auth_gssapi_client Plugin

gssapi implementation

authentication plugin data format :

  • string serverPrincipalName (UTF-8 format)

  • string mechanisms (UTF-8 format)

Client must exchange packet with server until having a mutual GSSAPI authentication. The only difference compared to standard client-server GSSAPI authentication is that exchanges contain standard protocol with packet headers.

client_ed25519 Plugin

The ed25519 plugin uses the Elliptic Curve Digital Signature Algorithm to securely store users' passwords and to authenticate users. It has been Implemented in the server since MariaDB 10.1.22.

See plugin description.

The server sends a random nonce that the client signs.

authentication plugin data format :

  • byte seed

Client response :

  • byte ed25519 encrypted password

parsec Plugin

authentication plugin data format :

  • string<32> server nonce

Client has to send an empty packet to request "ext-salt".

Format of ext-salt is

  • string<1> 'P' (denotes KDF algorithm = PBKDF2)

  • byte<1> iteration factor. number of iterations correspond to 1024 << iteration factor (0x0 means 1024, 0x1 means 2048, etc.)

  • byte salt

client must then :

  • generate derived key = hash password with PBKDF2 ( sha512 digest) with iteration number and salt from ext-salt.

  • generate a client 32 bytes nonce

  • generate the signature with ed25519 of an array concatenation of server nonce + client nonce with the generated derived key as private key.

Client response :

  • byte<32> client nonce

  • byte<64> signature

Capabilities

Server and Client have different capabilities, here is the possibles values. client with capabilities CLIENT_MYSQL + CONNECT_WITH_DB will have a value of 9 (1 + 8).

CLIENT_MYSQL

1

Set by older MariaDB versions. MariaDB 10.2 leaves this bit unset to permit MariaDB identification and indicate support for extended capabilities. (MySQL named this CLIENT_LONG_PASSWORD)

FOUND_ROWS

2

CONNECT_WITH_DB

8

One can specify db on connect

COMPRESS

32

Can use compression protocol

LOCAL_FILES

128

Can use LOAD DATA LOCAL

IGNORE_SPACE

256

Ignore spaces before '('

CLIENT_PROTOCOL_41

1 << 9

4.1 protocol

CLIENT_INTERACTIVE

1 << 10

SSL

1 << 11

Can use SSL

TRANSACTIONS

1 << 13

SECURE_CONNECTION

1 << 15

4.1 authentication

MULTI_STATEMENTS

1 << 16

Enable/disable multi-stmt support

MULTI_RESULTS

1 << 17

Enable/disable multi-results

PS_MULTI_RESULTS

1 << 18

Enable/disable multi-results for PrepareStatement

PLUGIN_AUTH

1 << 19

Client supports plugin authentication

CONNECT_ATTRS

1 << 20

Client send connection attributes

PLUGIN_AUTH_LENENC_CLIENT_DATA

1 << 21

Enable authentication response packet to be larger than 255 bytes

CLIENT_CAN_HANDLE_EXPIRED_PASSWORDS

1 << 22

Client can handle expired passwords

CLIENT_SESSION_TRACK

1 << 23

Enable/disable session tracking in OK_Packet

CLIENT_DEPRECATE_EOF

1 << 24

EOF_Packet deprecation : * OK_Packet replace EOF_Packet in end of Resulset when in text format * EOF_Packet between columns definition and resultsetRows is deleted

CLIENT_OPTIONAL_RESULTSET_METADATA

1 << 25

Not use for MariaDB

CLIENT_ZSTD_COMPRESSION_ALGORITHM

1 << 26

Support zstd protocol compression

CLIENT_CAPABILITY_EXTENSION

1 << 29

Reserved for future use. (Was CLIENT_PROGRESS Client support progress indicator before 10.2)

CLIENT_SSL_VERIFY_SERVER_CERT

1 << 30

Client verify server certificate. deprecated, client have options to indicate if server certifiate must be verified

CLIENT_REMEMBER_OPTIONS

1 << 31

MARIADB_CLIENT_PROGRESS

1 << 32

Client support progress indicator (since 10.2)

MARIADB_CLIENT_COM_MULTI

1 << 33

Permit COM_MULTI protocol

MARIADB_CLIENT_STMT_BULK_OPERATIONS

1 << 34

Permit bulk insert

MARIADB_CLIENT_EXTENDED_METADATA

1 << 35

Add extended metadata information

MARIADB_CLIENT_CACHE_METADATA

1 << 36

Permit skipping metadata

MARIADB_CLIENT_BULK_UNIT_RESULTS

1 << 37

when enable, indicate that Bulk command can use STMT_BULK_FLAG_SEND_UNIT_RESULTS flag that permit to return a result-set of all affected rows and auto-increment values

Native Password Authentication

The 20 byte string 'seed' is calculated by concatenating scramble first part (8 bytes) and scramble second part from Initial handshake packet. After that, the client calculates a password hash using the password and seed by using ^ (bitwise xor), + (string concatenation) and SHA1 as follows:

SHA1( passwd) ^ SHA1( seed + SHA1( SHA1( passwd ) ) )

This page is licensed: CC BY-SA / Gnu FDL

sha256_password Plugin

SHA256 authentication possible exchanges:

  • if connection use SSL (SSLRequest Packet sent)

    • client send a clear password answer

  • else

    • if client doesn't know server RSA public key

      • client sends a public key request

      • server sends a public key response

    • client sends an RSA encrypted password

    • ends with server sending either OK_Packet , ERR_Packet

Client clear password answer

  • string password without encryption

Public key request

  • byte<1> fixed 0x01 value

Public key response

  • byte<1> fixed 0x01 value

  • byte public key data

RSA encrypted password

  • byte<256> RSA encrypted password

RSA encrypted value of XOR(password, seed) using server public key (RSA_PKCS1_OAEP_PADDING).

This page is licensed: CC BY-SA / Gnu FDL

2 - Text Protocol

Text protocol commands are sent from client to server. The text protocol has a less comprehensive format than the binary protocol, which is used by prepared statements only

COM_CHANGE_USER

COM_CHANGE_USER resets the connection and re-authenticates with the given credentials. The packet is identical to the authentication packet in the connection handshake.

Fields

  • int<1> 0x11 : COM_CHANGE_USER header

  • string username

  • if (server_capabilities & CLIENT_SECURE_CONNECTION)

    • int<1> length of authentication response

    • string authentication response

  • else

    • string authentication response

  • string default schema name

  • int<2> client character collation

  • if (server_capabilities & CLIENT_PLUGIN_AUTH)

    • string authentication plugin name

  • if (server_capabilities & CLIENT_CONNECT_ATTRS)

    • int size of connection attributes

    • loop:

      • string key

      • string value

Response

Server response is like connection authentication :

  • An OK packet in case of success OK_Packet.

  • An error packet in case of error ERR_Packet.

  • Authentication switch

    • If the client or server doesn't have PLUGIN_AUTH capability:

      • Server sends 0xFE byte .

      • Client sends old_password.

    • else

      • Server sends Authentication switch request.

      • Client may have many exchanges with the server according to the Plugin.

    • Authentication switch ends with server sending either OK_Packet or ERR_Packet.

This page is licensed: CC BY-SA / Gnu FDL

COM_CREATE_DB

Warning: This command is deprecated and not used by MariaDB connectors any more. Please use the SQL statements CREATE SCHEMA or CREATE DATABASE instead.

This page is licensed: CC BY-SA / Gnu FDL

COM_DEBUG

The COM_DEBUG command forces the server to dump debug information to stdout. It requires super privileges.

Fields

  • int<1> 0xOD : COM_DEBUG Header

Response

EOF Packet

This page is licensed: CC BY-SA / Gnu FDL

COM_DROP_DB

Warning: This command is deprecated and not used by MariaDB connectors any more. Please use the SQL statements DROP SCHEMA or DROP DATABASE instead.

This page is licensed: CC BY-SA / Gnu FDL

COM_FIELD_LIST

Warning: This command is deprecated and not used by MariaDB connectors any more. Please use the SQL statements SHOW COLUMNS or SELECT FROM INFORMATION_SCHEMA.COLUMNS instead.

Fields

  • int<1> 0x04 : COM_FIELD_LIST Header

Response

  • n resultset row

  • EOF_Packet

This page is licensed: CC BY-SA / Gnu FDL

COM_INIT_DB

COM_INIT_DB is used to specify the default schema for the connection.

Fields

  • int<1> 0x02 : COM_INIT_DB Header

  • string schema name

Response

ERR_Packet or OK_Packet

Example

06 00 00 00 02 74 65 73 74 63                    .....testc

This page is licensed: CC BY-SA / Gnu FDL

COM_PING

COM_PING permits sending a packet containing one byte to check that the connection is active.

Fields

  • int<1> 0x0e : COM_PING Header

Response

OK_Packet

This page is licensed: CC BY-SA / Gnu FDL

COM_PROCESS_KILL

Forces the server to terminate a specified connection.

Fields

  • int<1> 0xC COM_PROCESS_KILL

  • int<4> process id

Response

OK Packet or ERR Packet

This page is licensed: CC BY-SA / Gnu FDL

COM_PROCESSLIST

Warning: This command is deprecated and not used by MariaDB connectors any more. Please use the SQL statement SHOW PROCESSLIST instead.

This page is licensed: CC BY-SA / Gnu FDL

COM_QUERY

With the COM_QUERY command, the client sends the server an SQL statement to be executed immediately.

Fields

  • int<1> 0x03 : COM_QUERY header

  • string SQL statement

The SQL statement should be properly escaped. The escape character is usually a backslash '' = 0x5c. However, if the status flag returned by the last OK Packet had NO_BACKSLASH_ESCAPES bit set then the escape character is a single quote(' = 0x60)

If the escape character is a backslash, the following characters are escaped:

  • single quote (' = 0x60)

  • back slash (\ = 0x5c)

  • double quote (" = 0x22)

  • null character (0x00)

If the escape character is a single quote, only the single quote (' = 0x60) can be escaped.

Response

The server can answer with 4 different responses that can be differentiated by the first byte (packet header):

  • 0xFF - ERR_Packet if any error occurs.

  • 0x00 - OK_Packet when query execution works without resultset.

  • 0xFB - LOCAL_INFILE Packet if the query was "LOCAL INFILE ...".

  • Or a Resultset, when the query returns results (in case of a SELECT query for example).

Example

1b 00 00 00 03 44 52 4f 50 20 54 41 42 4c 45 20

.....DROP TABLE

49 46 20 45 58 49 53 54 53 20 62 75 6c 6b 31

IF EXISTS bulk1

This page is licensed: CC BY-SA / Gnu FDL

COM_QUIT

Using the COM_QUIT command, the client tells the server that the connection should be terminated.

Fields

  • int<1> 0x01 : COM_QUIT Header

Response

Server terminates connection.

This page is licensed: CC BY-SA / Gnu FDL

COM_RESET_CONNECTION

COM_RESET_CONNECTION Resets a connection without re-authentication.

This will :

  • rollback any open transaction

  • reset transaction isolation level

  • reset session variables

  • delete user variables

  • remove temporary tables

  • remove all PREPARE statement

Database will NOT be reset to initial value.

Fields

  • int<1> 0x1f : COM_RESET_CONNECTION Header

Response

ERR_Packet or OK_Packet

This page is licensed: CC BY-SA / Gnu FDL

COM_SET_OPTION

Enables or disables server option.

Fields

  • int<1> 0x1B COM_SET_OPTION

  • int<2> option

Options

Constant

Value

MYSQL_OPTION_MULTI_STATEMENTS_ON

0

MYSQL_OPTION_MULTI_STATEMENTS_OFF

1

Response

EOF Packet on success or ERR packet.

This page is licensed: CC BY-SA / Gnu FDL

COM_SHUTDOWN

Shuts down the server. To execute this command the SHUTDOWN privilege is required.

Fields

  • int<1> 0x0A COM_SHUTDOWN

  • int<1> option

Options

Constant

Value

SHUTDOWN_DEFAULT

0

Response

OK Packet or ERR packet.

This page is licensed: CC BY-SA / Gnu FDL

COM_SLEEP

This command is used inside the server only.

Direction

used inside server only

Fields

  • int<1> 0x00 : COM_SLEEP header

Response

ERR_Packet

This page is licensed: CC BY-SA / Gnu FDL

COM_STATISTICS

Get internal server statistics.

Fields

  • int<1> 0x09 : COM_STATISTICS Header

Response

string Human readable string

This page is licensed: CC BY-SA / Gnu FDL

3 - Binary Protocol (Prepared Statements)

The binary protocol defines a non-text-based (binary) format that is used instead of the text protocol, only for prepared statements

COM_STMT_CLOSE

Closes a previously prepared statement.

Direction

Client to server.

Implemented by

  • mysql_stmt_close()

  • mysql_stmt_prepare()

Fields

  • int<1> 0x19 COM_STMT_CLOSE header

  • int<4> Statement id

Example

05 00 00 00 19 04 00 00 00

Response

No response from server.

This page is licensed: CC BY-SA / Gnu FDL

COM_STMT_BULK_EXECUTE

Executes a bulk insert of a previously prepared statement.

A command that returns a resultset will return an error (Error packet).

Direction

Client to server.

Fields

  • int<1> 0xfa : COM_STMT_BULK_EXECUTE header

  • int<4> statement id

  • int<2> bulk flags

  • if (bulk_flag & SEND_TYPES_TO_SERVER)

    • for each parameter :

      • byte<1>: field type

      • byte<1>: parameter type flag

  • until end of packet

    • for each parameter (i.e param_count times)

      • byte<1>: parameter indicator

      • if indicator == NONE

        • byte : binary parameter value

Flags

bulk flags

SEND_UNIT_RESULTS

64

Return generated affected rows and auto-increment IDs as a resultset (only when server supports MARIADB_CLIENT_BULK_UNIT_RESULTS capability)

SEND_TYPES_TO_SERVER

128

Send types to server

parameter type flag:

128

unsigned

parameter indicator

0

NONE

Value follow

1

NULL

Value is null

2

DEFAULT

For INSERT/UPDATE, value is default

3

IGNORE

Value is default for insert, Is ignored for update

COM_STMT_BULK_EXECUTE response

The server can answer with 3 different responses:

  • 0xff: ERR_Packet if any errors occur.

  • 0x00: OK_packet when query execution works without resultset.

  • a resultset containing affected rows and auto-increment IDs when bulk flag SEND_UNIT_RESULTS is set

This page is licensed: CC BY-SA / Gnu FDL

COM_STMT_EXECUTE

Executes a previously prepared statement.

If specific data is large, it can be sent separately prior to this command (see COM_STMT_SEND_LONG_DATA).

If a statement is re-executed without changing the bind types, the types do not need to be sent to the server again.

Direction

Client to server.

Fields

  • int<1> 0x17 : COM_STMT_EXECUTE header

  • int<4> statement id

  • int<1> flags:

  • int<4> Iteration count (always 1)

  • if (param_count > 0)

    • byte<(param_count + 7)/8> null bitmap

    • byte<1>: send type to server (0 / 1)

    • if (send type to server)

      • for each parameter :

        • byte<1>: field type

        • byte<1>: parameter flag

    • for each parameter (i.e param_count times)

      • if parameter is not null

        • byte binary parameter value

Statement Id

Statement id is the identifier of the prepared statement (from COM_STMT_PREPARE answer)

Specific "-1" statement id value

Since MariaDB server version 10.2, value -1 (0xFFFFFFFF) can be used to indicate to use the last statement prepared on current connection if no COM_STMT_PREPARE has fail since.

This permit pipelining :

  • send COM_STMT_PREPARE + COM_STMT_EXECUTE with statement id -1

  • read COM_STMT_PREPARE + COM_STMT_EXECUTE response

In case COM_STMT_PREPARE returns an error, COM_STMT_EXECUTE will return an error that statement id -1 is unknown. This permits to avoid much of the network latency.

Flag

0

no cursor

1

read only

2

cursor for update

4

scrollable cursor

Cursors

If the flags of the COM_STMT_EXECUTE request a cursor to be opened, the returned result will only contain the column definitions and the EOF that terminates it and the resultset rows are fetched using separate COM_STMT_FETCH commands.

Whether a cursor is actually opened is indicated by the SERVER_STATUS_CURSOR_EXISTS bit in the first EOF packet in the response to the COM_STMT_EXECUTE. If it is not set in, no cursor is opened and a normal resultset is returned.

Parameter flag

parameter type flag byte:

128

unsigned

NULL-Bitmap

The NULL-Bitmap indicates if parameters are null (one bit per parameter). If the parameter is NULL, the bit is set in the bitmap and the parameter value is not sent.

The size in bytes of the NULL-bitmap can be calculated with: (parameter number + 7) / 8

Binary parameter encoding

The encoding of the COM_STMT_EXECUTE parameters are the same as the encoding of the binary resultsets.

COM_STMT_EXECUTE response

The server can answer with 3 different responses:

  • 0xff: ERR_Packet if any errors occur.

  • 0x00: OK_packet when query execution works without resultset.

  • one (or more) Resultset, when query execution return rows (in case of SELECT query for example).

This page is licensed: CC BY-SA / Gnu FDL

COM_STMT_FETCH

Fetch rows from a prepared statement.

A COM_STMT_EXECUTE with a non-zero cursor flag must have been successfully executed before any COM_STMT_FETCH commands can be executed.

Fields

  • int<1> 0x1C COM_STMT_FETCH header.

  • int<4> statement id.

  • int<4> number of rows to fetch.

Response

Returns one or more binary result set rows followed by an EOF packet.

This page is licensed: CC BY-SA / Gnu FDL

COM_STMT_PREPARE

Prepares a statement on the server.

Not all statements can be prepared. See PREPARE for a list of statements that can be prepared.

Implemented by

  • mariadb_stmt_execute_direct()

  • mysql_stmt_prepare()

Fields

  • int<1> 0x16 COM_STMT_PREPARE header

  • string SQL Statement

Example

1F 00 00 00 16 53 45 4C 45 43 54 20 2A 20 46 52  .....SELECT * FR
1F 4D 20 74 65 73 74 5F 62 69 6E 64 5F 72 65 73  OM test_bind_res
75 6C 74                                         ult

COM_STMT_PREPARE response

If anything goes wrong, the server will send an ERR_Packet. If the command succeeds, different packets will be received:

  • COM_STMT_PREPARE_OK

  • if number of parameters (count of '?' placeholders) > 0

    • for each parameter:

      • column definition packet

    • if !DEPRECATE_EOF eof_packet

  • if prepared statement returns result set and number of result set columns > 0

    • for each column:

      • column definition packet

    • if !DEPRECATE_EOF eof_packet

COM_STMT_PREPARE_OK

  • int<1> 0x00 COM_STMT_PREPARE_OK header

  • int<4> statement id

  • int<2> number of columns in the returned result set (or 0 if statement does not return result set)

  • int<2> number of prepared statement parameters ('?' placeholders)

  • string<1> -not used-

  • int<2> number of warnings

This page is licensed: CC BY-SA / Gnu FDL

COM_STMT_RESET

Resets a prepared statement on the client and server to state after preparing.

Fields

  • int<1> 0x1A COM_STMT_RESET header

  • int<4> Statement id

Response

ERR_Packet or OK_Packet

Example

05 00 00 00 1A 04 00 00 00

This page is licensed: CC BY-SA / Gnu FDL

COM_STMT_SEND_LONG_DATA

When data for a specific column is large, it can be sent separately to avoid the limitation of max_allowed_packet (see packet splitting).

Multiple COM_STMT_SEND_LONG_DATA commands with the same column id will append the data. COM_STMT_SEND_LONG_DATA will be sent before COM_STMT_EXECUTE.

Fields

  • int<1> 0x18 COM_STMT_SEND_LONG_DATA header

  • int<4> statement id

  • int<2> parameter number

  • byte data

Response

Server doesn't send response.

This page is licensed: CC BY-SA / Gnu FDL

Server Response Packets (Binary Protocol)

The following response packets will be sent from the server to client in binary protocol only. They may be mixed with packets from the text protocol, like OK_PACKET, PACKET_METADATA or ERR_PACKET.

PACKET_BINDATA

In contrast to the text protocol, the binary protocol transfers the data according to the format of the field types returned in PACKET_METADATA.

Field type
Representation

MYSQL_TYPE_BIT

str_LEC

MYSQL_TYPE_BLOB

str_LEC

MYSQL_TYPE_DATE MYSQL_TYPE_DATETIME MYSQL_TYPE_TIMESTAMP

int_11 (default)int_7 (no microseconds)int_4 (no time values)int_0 (no values)

MYSQL_TYPE_DECIMAL

str_LEC

MYSQL_TYPE_DOUBLE

int_8

MYSQL_TYPE_ENUM

str_LEC

MYSQL_TYPE_FLOAT

int_4

MYSQL_TYPE_GEOMETRY

str_LEC

MYSQL_TYPE_INT24

int_4

MYSQL_TYPE_JSON

str_LEC

MYSQL_TYPE_LONGLONG

int_8

MYSQL_TYPE_LONG_BLOB

str_LEC

MYSQL_TYPE_LONG

int_4

MYSQL_TYPE_MEDIUM_BLOB

str_LEC

MYSQL_TYPE_NEWDECIMAL

str_LEC

MYSQL_TYPE_NULL

stored in bitmap

MYSQL_TYPE_SET

str_LEC

MYSQL_TYPE_STRING

str_LEC

MYSQL_TYPE_SHORT

int_2

MYSQL_TYPE_TINY_BLOB

str_LEC

MYSQL_TYPE_TINY

int_1

MYSQL_TYPE_VARCHAR

str_LEC

MYSQL_TYPE_VAR_STRING

str_LEC

MYSQL_TYPE_YEAR

int_4

Fields

(column_count + 7)/8   null bitmap
while (!eof) {
  for (i=0; i < column_count; i++)
  {
    data (length depends on the data type)
  }
}

This page is licensed: CC BY-SA / Gnu FDL

4 - Server Response Packets

For most commands which the client sends to the server, the server returns the following response packets

EOF_Packet

The eof packet marks the end of a resultset and returns status and warnings.

When testing for an EOF packet, the packet size must be less than 9 bytes in length. Resultset can send data that begins with a 0xfe byte, but then the packet length will be greater than 9.

Fields

  • int<1> 0xfe : EOF header

  • int<2> warning count

  • int<2> server status

This page is licensed: CC BY-SA / Gnu FDL

ERR_Packet

ERR_Packet

ERR_Packet indicates that an error occured.

Fields

  • int<1> ERR_Packet header = 0xFF

  • int<2> error code. see error list

  • if (errorcode == 0xFFFF) /* progress reporting */

    • int<1> stage

    • int<1> max_stage

    • int<3> progress

    • string progress_info

  • else

    • if (next byte = '#')

      • string<1> sql state marker '#'

      • string<5>sql state

      • string human-readable error message

    • else

      • string human-readable error message

Note that the ERR packet is supposed to send a server error to the client. In particular, all error codes in the range 2000 to 2999 and 5000 to 5999 (inclusive) are reserved for client errors and an ERR packet with such an error code will be considered malformed.

See also

"Progress reporting"

This page is licensed: CC BY-SA / Gnu FDL

OK_Packet

OK_Packet is sent by the server to the client and indicates a successful completion of a command sent by the client before.

Direction

Server to client.

Fields

  • int<1> 0x00 : OK_Packet header or (0xFE if CLIENT_DEPRECATE_EOF is set)

  • int affected rows

  • int last insert id

  • int<2> server status

  • int<2> warning count

  • if packet has more data

    • string info

    • if (status flags & SERVER_SESSION_STATE_CHANGED) and session_tracking_supported (see CLIENT_SESSION_TRACK)

      • string session state info

The length-encoded info string is not always included in the packet. Check the length of the packet to detect if there is data after the warning count. For the first OK_Packet in the connection it contains (if present) the SSL certificate verification signature. For the following OK_Packets it contains (if present) various human-readable information.

Server status flag

Values of server status flag

SERVER_STATUS_IN_TRANS

1

A transaction is currently active

SERVER_STATUS_AUTOCOMMIT

2

Autocommit mode is set

SERVER_MORE_RESULTS_EXISTS

8

More results exists (more packets will follow)

SERVER_QUERY_NO_GOOD_INDEX_USED

16

Set if EXPLAIN would've shown Range checked for each record

SERVER_QUERY_NO_INDEX_USED

32

The query did not use an index

SERVER_STATUS_CURSOR_EXISTS

64

When using COM_STMT_FETCH, indicate that current cursor still has result

SERVER_STATUS_LAST_ROW_SENT

128

When using COM_STMT_FETCH, indicate that current cursor has finished to send results

SERVER_STATUS_DB_DROPPED

1<<8

Database has been dropped

SERVER_STATUS_NO_BACKSLASH_ESCAPES

1<<9

Current escape mode is "no backslash escape"

SERVER_STATUS_METADATA_CHANGED

1<<10

A DDL change did have an impact on an existing PREPARE (an automatic reprepare has been executed)

SERVER_QUERY_WAS_SLOW

1<<11

The query was slower than long_query_time

SERVER_PS_OUT_PARAMS

1<<12

This resultset contain stored procedure output parameter

SERVER_STATUS_IN_TRANS_READONLY

1<<13

Current transaction is a read-only transaction

SERVER_SESSION_STATE_CHANGED

1<<14

Session state change. See Session change type for more information

Session state info

  • while packet has remaining data

    • int<1> session change type

    • if (session-change-type != SESSION_TRACK_STATE_CHANGE)

      • int total length

    • string session data's change

Session change type

0

SESSION_TRACK_SYSTEM_VARIABLES

1

SESSION_TRACK_SCHEMA

2

SESSION_TRACK_STATE_CHANGE

3

SESSION_TRACK_GTIDS

4

SESSION_TRACK_TRANSACTION_CHARACTERISTICS

5

SESSION_TRACK_TRANSACTION_STATE

session data's change

Each type of data has his own kind of format :

SESSION_TRACK_SCHEMA

  • string new current schema

SESSION_TRACK_SYSTEM_VARIABLES

while there is remaining data :

  • string variable data

for each variable data :

  • string variable name

  • string variable value

Possible tracked variables list is tracked by session_track_system_variables special variable value description:

  • redirect_url: format is mariadb/mysql:[<user>[:<password>]@]<host>[:<port>]/[<db>[?<opt1>=<value1>[&<opt2>=<value2>]]]. Possible options:

    • ttl : cache timeout in ms to remember redirection, in order to reconnect directly to new host. 0=no caching

SESSION_TRACK_STATE_CHANGE

indicates if session state changes occured. The value is represented as "1".

  • string "1" if session state tracking was enabled

SESSION_TRACK_GTIDS

This tracker is not implemented by MariaDB.

SESSION_TRACK_TRANSACTION_CHARACTERISTICS

  • string Transaction characteristics

The transaction characteristics is the set of SQL statements that reproduces the type and state of the current transaction. It can consist of the following SQL statements:

SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;
SET TRANSACTION ISOLATION LEVEL READ COMMITTED;
SET TRANSACTION ISOLATION LEVEL REPEATABLE READ;
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;
START TRANSACTION;
START TRANSACTION WITH CONSISTENT SNAPSHOT;
START TRANSACTION WITH CONSISTENT SNAPSHOT, READ ONLY;
START TRANSACTION WITH CONSISTENT SNAPSHOT, READ WRITE;
SET TRANSACTION READ ONLY;
SET TRANSACTION READ WRITE;
XA START <XA specification>;

SESSION_TRACK_TRANSACTION_STATE

  • string Transaction state string

The transaction state string is always 8 characters long. The characters, in order, are:

  1. No transaction: _ Explicit transaction: T Implicit transaction: I

  2. Transaction read safe: _ Transaction read unsafe: r

  3. Unknown transaction type: _ Read-only transaction: R

  4. Transaction write safe: _ Transaction write unsafe: w

  5. Unknown transaction type: _ Read-write transaction: W

  6. Transaction statement safe: _ Transaction statement unsafe: s

  7. Transaction does not have resultsets: _ Transaction with resultsets: S

  8. No locked tables: _ Tables have been locked: L

Notes

  • Session tracking is supported from MariaDB 10.2. To determine if session tracking is enabled, check if the CLIENT_SESSION_TRACK flag is set in server_capabilities.

This page is licensed: CC BY-SA / Gnu FDL

LOCAL_INFILE Packet

LOCAL_INFILE Packet

If the client sends a LOAD DATA LOCAL INFILE command via com_query, the server responds with LOCAL_INFILE_Packet to tell the client to send a specified file to the server.

Fields

  • int<1> 0xFB : LOCAL_INFILE header

  • string filename

Client response

The client sends the file as the packet body. If the file is large, the contents are sent in multiple separate packets. After the file is sent, the client must send an empty packet to indicate that no more data will follow.

Once the client has finished sending the file, the server will respond with an OK_packet or an ERR_packet.

This page is licensed: CC BY-SA / Gnu FDL

Result Set Packets

MariaDB Server sends the following packets as part of a result set: A resultset consists of different packets:

  • Resultset metadata

    • 1 Column count packet

  • if not (MARIADB_CLIENT_CACHE_METADATA capability set) OR (send metadata == 1)

    • for each column (i.e column_count times)

      • Column Definition packet

  • if not (CLIENT_DEPRECATE_EOF capability set) EOF_Packet

  • n resultset row

  • if error

    • ERR_Packet

  • else

    • if CLIENT_DEPRECATE_EOF capability

      • OK_Packet with a 0xFE header

    • else EOF_Packet

It would be unsafe to assume that any packet with a 0xFE header is an OK packet (OK_Packet) or an EOF packet (EOF_Packet), because result-set row packets (ResultsetRow) can also begin with 0xFE when using the text protocol with a field length greater than 0xFFFFFF. To safely confirm that a packet with a 0xFE header is an OK packet (OK_Packet) or an EOF packet (EOF_Packet), you must also check that the packet length is less than 0xFFFFFF.

Column Count Packet

The column count packet describes the number of columns in the result set. It uses the following format:

  • int column count

  • if (MARIADB_CLIENT_CACHE_METADATA capability set)

    • int<1> metadata follows (0 / 1)

The metadata indicator byte is only present if both the client and the server declare the MARIADB_CLIENT_CACHE_METADATA capability.

If the metadata byte is set to 1, the normal metadata will follow the column definitions. If the metadata byte is set to 0, the Column Count Packet is immediately followed by the second EOF packet (EOF_Packet) or the resultset rows if the CLIENT_DEPRECATE_EOF capability is set.

Column Definition Packet

A column definition packet describes a column in the result set. It uses the following format:

  • string catalog (always 'def')

  • string schema

  • string table alias

  • string table

  • string column alias

  • string column

  • if extended type supported (see MARIADB_CLIENT_EXTENDED_METADATA )

    • string extended metadata

  • int length of fixed fields (=0xC)

  • int<2> character set number

  • int<4> max. column size

  • int<1> Field types

  • int<2> Field detail flag

  • int<1> decimals

  • int<2> - unused -

Field types

The column type field in the column definition packet describes the base type of the column. It also indicates how the values are encoded for COM_STMT_EXECUTE parameters and binary resultset rows.

Value
Protocol Column Type
Encoding

0

MYSQL_TYPE_DECIMAL

byte encoding

1

MYSQL_TYPE_TINY

TINYINT Binary encoding

2

MYSQL_TYPE_SHORT

SMALLINT Binary encoding

3

MYSQL_TYPE_LONG

INTEGER Binary encoding

4

MYSQL_TYPE_FLOAT

FLOAT Binary encoding

5

MYSQL_TYPE_DOUBLE

DOUBLE Binary encoding

6

MYSQL_TYPE_NULL

Not used, nullness is indicated by the NULL-bitmap in the result

7

MYSQL_TYPE_TIMESTAMP

TIMESTAMP Binary encoding

8

MYSQL_TYPE_LONGLONG

BIGINT Binary encoding

9

MYSQL_TYPE_INT24

INTEGER Binary encoding

10

MYSQL_TYPE_DATE

TIMESTAMP Binary encoding

11

MYSQL_TYPE_TIME

TIME Binary encoding

12

MYSQL_TYPE_DATETIME

TIMESTAMP Binary encoding

13

MYSQL_TYPE_YEAR

SMALLINT Binary encoding

14

MYSQL_TYPE_NEWDATE

byte encoding

15

MYSQL_TYPE_VARCHAR

byte encoding

16

MYSQL_TYPE_BIT

byte encoding

17

MYSQL_TYPE_TIMESTAMP2

Used only in the replication protocol

18

MYSQL_TYPE_DATETIME2

Used only in the replication protocol

19

MYSQL_TYPE_TIME2

Used only in the replication protocol

245

MYSQL_TYPE_JSON

byte encoding (only used with MySQL, MariaDB uses MYSQL_TYPE_STRING for JSON)

246

MYSQL_TYPE_NEWDECIMAL

byte encoding

247

MYSQL_TYPE_ENUM

byte encoding

248

MYSQL_TYPE_SET

byte encoding

249

MYSQL_TYPE_TINY_BLOB

byte encoding

250

MYSQL_TYPE_MEDIUM_BLOB

byte encoding

251

MYSQL_TYPE_LONG_BLOB

byte encoding

252

MYSQL_TYPE_BLOB

byte encoding

253

MYSQL_TYPE_VAR_STRING

byte encoding

254

MYSQL_TYPE_STRING

byte encoding

255

MYSQL_TYPE_GEOMETRY

byte encoding

Field Details Flag

The column details flag describes certain column attributes and whether certain column options are set.

It is a bitmask with the following flags:

Flag Value
Flag Name
Flag Description

1

NOT_NULL

field cannot be null

2

PRIMARY_KEY

field is a primary key

4

UNIQUE_KEY

field is unique

8

MULTIPLE_KEY

field is in a multiple key

16

BLOB

is this field a Blob

32

UNSIGNED

is this field unsigned

64

ZEROFILL_FLAG

is this field a zerofill

128

BINARY_COLLATION

whether this field has a binary collation

256

ENUM

Field is an enumeration

512

AUTO_INCREMENT

field auto-increment

1024

TIMESTAMP

field is a timestamp value

2048

SET

field is a SET

4096

NO_DEFAULT_VALUE_FLAG

field doesn't have default value

8192

ON_UPDATE_NOW_FLAG

field is set to NOW on UPDATE

32768

NUM_FLAG

field is num

The BLOB flag cannot be used to determine if a column has binary data, because BINARY and VARBINARY columns are treated as strings, instead of blobs.

The BINARY_COLLATION flag can be used to determine if a string column has binary data.

Extended metadata

This extended column type information can be used to find out more specific details about the column type.

For example:

  • For a POINT column, the column type field will be MYSQL_TYPE_GEOMETRY, but the extended type will indicate 'point'.

  • For a JSON column, the column type field will be MYSQL_TYPE_STRING, but the extended type will indicate 'json'.

  • while string has data

    • int<1> data type: 0x00:type, 0x01: format

    • string value

This page is licensed: CC BY-SA / Gnu FDL

Resultset row

A resultset row represents a database resultset unit, which is usually generated by executing a statement that queries the database. Using COM_STMT_EXECUTE the resultsetrow will be in binary format, others in text format.

Text resultset row

  • for each column

    • string column data

The byte representation of the string according to client character collation.

Binary resultset row

  • byte<1> 0x00 header

  • byte<(number_of_columns + 7) / 8> NULL-Bitmap

  • for each column

    • if column value is not null

      • if MYSQL_TYPE_DOUBLE type : DOUBLE Binary encoding

      • if MYSQL_TYPE_LONGLONG type : BIGINT Binary encoding

      • if MYSQL_TYPE_INTEGER type : INTEGER Binary encoding

      • if MYSQL_TYPE_MEDIUMINT type : MEDIUMINT Binary encoding

      • if MYSQL_TYPE_FLOAT type : FLOAT Binary encoding

      • if MYSQL_TYPE_SMALLINT type : SMALLINTBinary encoding

      • if MYSQL_TYPE_YEAR type : YEAR Binary encoding

      • if MYSQL_TYPE_TINYINT type : TINYINT Binary encoding

      • if MYSQL_TYPE_DATE type : DATE Binary encoding

      • if MYSQL_TYPE_TIMESTAMP type : TIMESTAMP Binary encoding

      • if MYSQL_TYPE_DATETIME type : TIMESTAMP Binary encoding

      • if MYSQL_TYPE_TIME type : TIME Binary encoding

      • if MYSQL_TYPE_NEWDECIMAL type : DECIMAL Binary encoding

      • if MYSQL_TYPE_TINY_BLOB, MYSQL_TYPE_MEDIUM_BLOB, MYSQL_TYPE_LONG_BLOB, MYSQL_TYPE_BLOB, MYSQL_TYPE_GEOMETRY,MYSQL_TYPE_STRING, MYSQL_TYPE_VARCHAR, MYSQL_TYPE_VAR_STRING): byte value

NULL-Bitmap values

The NULL-Bitmap indicates if a parameter for a column is null (one bit per parameter) beginning with the 3rd bit. NULL-bitmap size is (number_of_columns + 7) / 8.

DECIMAL Binary encoding.

DECIMAL has no fixed size, so will be encoded as string. An DECIMAL(10,2) with a value of -15.5 is stored as

06 45 49 53 46 53 48      . - 1 5 . 5 0

DOUBLE binary encoding

DOUBLE is the IEEE 754 floating-point value in Little-endian format on 8 bytes.

BIGINT binary encoding

BIGINT is the value in Little-endian format on 8 bytes. Signed is defined by the Column field detail flag.

INTEGER binary encoding

INTEGER is the value in Little-endian format on 4 bytes. Signed is defined by the Column field detail flag.

MEDIUMINT binary encoding

MEDIUMINT is similar to INTEGER binary encoding, even if MEDIUM int is 3-bytes encoded server side. (Last byte will always be 0x00).

FLOAT binary encoding

FLOAT is the IEEE 754 floating-point value in Little-endian format on 4 bytes.

SMALLINT binary encoding

SMALLINT is the value in Little-endian format on 2 bytes. Signed is defined by the Column field detail flag.

YEAR binary encoding

YEAR uses the same format as SMALLINT.

TINYINT binary encoding

TINYINT is the value of 1 byte. Signed is defined by the Column field detail flag.

DATE binary encoding

DATE uses the same format as TIMESTAMP binary encoding, with a data length of 0 for the special '0000-00-00' value and 4 for the standard year/month/day format

TIMESTAMP binary encoding

Data is encoded in 8 bytes without fractional seconds, 12 bytes with fractional seconds.

byte position

description

1

data length : 0 for special '0000-00-00 00:00:00' value. 4 with year + month + day of month only 7 for timestamps without fractional seconds 11 with fractional seconds

2-3

year on 2 bytes little-endian format

4

Month ( 1=january)

5

days of month

6

hour of day (0 if DATE type)

7

minutes (0 if DATE type)

8

secondes (0 if DATE type)

9-12

micro-second on 4 bytes little-endian format (only if data-length is > 7)

TIME binary encoding

Data is encoded in 9 bytes without fractional seconds, 13 bytes with fractional seconds.

  • int<1> data length: 0 for special '00:00:00' value, 8 without fractional seconds, 12 with fractional seconds

  • if data length > 0

    • int<1> 0 for positive time, 1 for negative time

    • int<4> days

    • int<1> hours

    • int<1> minutes

    • int<1> seconds

    • if data length > 8

      • int<4> microseconds

This page is licensed: CC BY-SA / Gnu FDL

Replication Protocol

The MariaDB replication protocol governs the replication of data from a master server to downstream slave servers

1-Binlog Events

Binary log events, or binlog events, are information about data modification made to a MariaDB server instance stored in the binary log files.

Log File Structure

Each log file contains:

  • a 4-byte magic number, followed by a series of events describing data modifications: The magic number bytes are 0xfe 0x62 0x69 0x6e = 0xfe 'b''i''n' (this is the BINLOG_MAGIC constant).

  • series of binlog events.

Event Content

Each event contains the 'header' followed by 'data bytes':

The header bytes provide information about

  • event type

  • creation time

  • which server created the event

  • flags and so forth

The data bytes provide information specific to the type of event.

Note that the first event, FORMAT_DESCRIPTION_EVENT at 'position' 4, is a descriptor event that describes the format used to write events in the file.

The remaining events are interpreted according to the version.

The final event is usually a log-rotation event ROTATE_EVENT that specifies the next binary log filename or a STOP_EVENT written during server shutdown.

Note: in case of a server crash there is no terminating event (no ROTATE nor STOP)

Example MariaDB 10.1 Binlog File (Hexdump -C $file_name)

fe 62 69 6e a4 85 9e 59  0f 8c 27 00 00 f5 00 00  |.bin...Y..'.....|
00 f9 00 00 00 00 00 04  00 31 30 2e 31 2e 32 34  |.........10.1.24|
2d 4d 61 72 69 61 44 42  00 6c 6f 67 00 00 00 00  |-MariaDB.log....|
00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00 00 00 00 00 00 00 00  00 00 00 a4 85 9e 59 13  |..............Y.|
38 0d 00 08 00 12 00 04  04 04 04 12 00 00 dd 00  |8...............|
04 1a 08 00 00 00 08 08  08 02 00 00 00 0a 0a 0a  |................|
00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
04 13 04 00 01 ab 5b a2  e0 a4 85 9e 59 a3 8c 27  |......[.....Y..'|
00 00 2b 00 00 00 24 01  00 00 00 00 01 00 00 00  |..+...$.........|
00 00 00 00 8c 27 00 00  00 0e 00 00 00 00 00 00  |.....'..........|
...
...

Example From mysqlbinlog

DELIMITER /*!*/;
# at 4
#170824  9:52:04 server id 10124  end_log_pos 249 CRC32 0xe0a25bab 	Start: binlog v 4, server v 10.1.24-MariaDB created 170824  9:52:04 at startup
ROLLBACK/*!*/;
BINLOG '
pIWeWQ+MJwAA9QAAAPkAAAAAAAQAMTAuMS4yNC1NYXJpYURCAGxvZwAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAACkhZ5ZEzgNAAgAEgAEBAQEEgAA3QAEGggAAAAICAgCAAAACgoKAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAEEwQAAatbouA=
'/*!*/;
# at 249
#170824  9:52:04 server id 10124  end_log_pos 292 CRC32 0xb6d8f0a8 	Gtid list [0-10124-3584]
# at 292
#170824  9:52:04 server id 10124  end_log_pos 334 CRC32 0xf2dc685f 	Binlog checkpoint log-bin.000011
# at 334
#170824  9:52:13 server id 10124  end_log_pos 376 CRC32 0xe958a0ae 	GTID 0-10124-3585 trans
...
...

This page is licensed: CC BY-SA / Gnu FDL

2-Binlog Event Header

All the binlog events stored in a binary log file have a common structure:

  • an event header

  • event data

Event Header Structure, 19 Bytes

  • uint<4> Timestamp (creation time)

  • uint<1> Event Type (type_code)

  • uint<4> Server_id (server which created the event)

  • uint<4> Event Length (header + data)

  • uint<4> Next Event position

  • uint<2> Event flags

Note: if CRC32 is in use, the Event Length is 4 bytes bigger in size. The 4 bytes CRC32 are written at the end of the event (just after the last 'data' byte).

Encrypted Binlog Events

For encrypted binlog events, only the event length is in plaintext and everything else is encrypted.

To decrypt the binlog event:

  • Store the event length in memory

  • Move the timestamp into the event length position

  • Decrypt the whole payload except the first four bytes

  • Move the timestamp back to its original position

  • Copy the original event length back to its position

Regardless of the cipher used to encrypt the binlogs, the encrypted data will be the same size as the original unencrypted event. For events that are encrypted in CBC mode and whose length is not a multiple of the cipher block size, the final partial block is encrypted using a form of residual block termination:

  • Encrypt the current IV of the binlog file in ECB mode

  • XOR the remaining bytes with the encrypted IV

Event Type

Hex
Event type description

0x02

QUERY_EVENT

0x03

STOP_EVENT

0x04

ROTATE_EVENT

0x10

XID_EVENT

0x0d

RAND_EVENT

0x0e

USER_VAR_EVENT

0x0f

FORMAT_DESCRIPTION_EVENT

0x13

TABLE_MAP_EVENT

0x1b

HEARTBEAT_LOG_EVENT

0xa0

ANNOTATE_ROWS_EVENT

0xa1

BINLOG_CHECKPOINT_EVENT

0xa2

GTID_EVENT

0xa3

GTID_LIST_EVENT

0xa4

START_ENCRYPTION_EVENT

0xa5

QUERY_COMPRESSED_EVENT

0xa6

WRITE_ROWS_COMPRESSED_V1

0xa7

UPDATE_ROWS_COMPRESSED_V1

0xa8

DELETE_ROWS_COMPRESSED_V1

0xa9

WRITE_ROWS_V1

0xaa

UPDATE_ROWS_V1

0xab

DELETE_ROWS_V1

Fake Events

These are generated on the fly, never written.

0x04

FAKE_ROTATE_EVENT

0xa3

FAKE_GTID_LIST_EVENT

Event Flag

Hex
Event flag description

0x0001

LOG_EVENT_BINLOG_IN_USE_FThis flag only makes sense for Format_description_log_event. It is set when the event is written, and reset when a binlog file is closed (yes, it's the only case when MySQL modifies already written part of binlog). Thus it is a reliable indicator that binlog was closed correctly.

0x0002

LOG_EVENT_FORCED_ROTATE_F(unused)

0x0004

LOG_EVENT_THREAD_SPECIFIC_F If the query depends on the thread (for example: TEMPORARY TABLE)

0x0008

LOG_EVENT_SUPPRESS_USE_F Suppress the generation of 'USE' statements before the actual statement. This flag should be set for any events that does not need the current database set to function correctly. Most notable cases are 'CREATE DATABASE' and 'DROP DATABASE'.

0x0010

LOG_EVENT_UPDATE_TABLE_MAP_VERSION_F (unused)

0x0020

LOG_EVENT_ARTIFICIAL_F Artificial events are created arbitarily and not written to binary log.These events should not update the master log position when slave SQL thread executes them.

0x0040

LOG_EVENT_RELAY_LOG_F Events with this flag set are created by slave IO thread and written to relay log

0x0080

LOG_EVENT_IGNORABLE_F For an event, 'e', carrying a type code, that a slave,'s', does not recognize, 's' will check 'e' forLOG_EVENT_IGNORABLE_F, and if the flag is set, then 'e'is ignored. Otherwise, 's' acknowledges that it has found an unknown event in the relay log.

0x0100

LOG_EVENT_NO_FILTER_F (no description yet)

0x0200

LOG_EVENT_MTS_ISOLATE_F (no description yet)

0x8000

LOG_EVENT_SKIP_REPLICATION_F Flag set by application creating the event (with @@skip_replication);the slave will skip replication of such eventsif --replicate-events-marked-for-skip is not set to REPLICATE.This is a MariaDB flag; we allocate it from the end of the available values to reduce risk of conflict with new MySQL flags.

Event Header example of FORMAT_DESCRIPTION_EVENT

This is the first event in the binlog file at pos 4

a4 85 9e 59 0f 8c 27 00  00 f5 00 00 00 f9 00 00  ...Y..'.........
00 00 00 04 00 31 30 2e  31 2e 32 34 2d 4d 61 72  .....10.1.24-Mar
69 61 44 42 00 6c 6f 67  00 00 00 00 00 00 00 00  iaDB.log....
...
...

Interpretation of First 19 Bytes of the Event (the Event Header)

  • a4 85 9e 59 [4] Timestamp => 59 9e 85 a4 => 1503561124 = 2017-08-24 09:52:04

  • 0f [1] Event Type = 0x0f = FORMAT_DESCRIPTION_EVENT

  • 8c 27 00 00 [4] Server_id => 00 00 27 8c = 10124

  • f5 00 00 00 [4] Event length => 00 00 00 f5 => 245

  • f9 00 00 00 [4] Next Event pos => 00 00 00 f9 => 249 (pos 4 + event size)

  • 00 00 [2] Event flags = 0

This page is licensed: CC BY-SA / Gnu FDL

3-Binlog Network Stream

The binary log events stored in a binary log file can be sent over the network in order to replicate data changes from the master server (where data changes are written in binary logs) to slave servers which will apply data changes into their own databases.

The MariaDB slave replication protocol consists of:

  • registration phase to master

  • events receiving (Master sending data when changes are available).

Note: This section is related to events sending only.

Binlog Network streams are requested with COM_BINLOG_DUMP and each Binlog Event is prepended with a status byte. The data sent over network is then MariaDB network protocol (4 bytes) + 1 byte status flag + event data.

MariaDB network protocol 4 bytes are:

  • uint<3> packet length (The sent binlog event can be up to 2^24 - 1 - 1 data bytes)

  • uint<1> packet sequence byte<1>(0 to 255)

Replication protocol status byte:

  • uint<1> OK (0) or ERR (ff) or End of File, EOF, (fe)

Note: due to the 1 byte status flag the effective data payload is event_size + 1 This means than an event of exact 16Mbytes (2 ^ 24 - 1) cannot be sent in one transmission: it requires 2 packets instead.

packet #n:   3 bytes length + sequence + status + [event_header + (event data - 1)]
packet #n+1: 3 bytes length + sequence + last byte of the event data.

Please note that the remaining bytes of a large event transmission are always sent without a status flag and binlog event header: just network packet header + data.

Example of an Event Transmission HEARTBEAT_LOG_EVENT

T 127.0.0.1:8808 -> 127.0.0.1:57157 [AP]
  23 00 00 04 00 00 00 00    00 1b 67 2b 00 00 22 00    '.........g+..&.
  00 00 ed 01 00 00 20 00    66 6f 6f 2d 62 69 6e 2e    ...... .log-bin.
  31 30 30 30 31 33 39                                  1000139

Network Replication Protocol, 5 Bytes

  • packet size [3] = 23 00 00 => 00 00 23 => 35 (ok byte + event size)

  • pkt sequence [1] = 04

  • OK indicator [1] = 0 (OK)

Heartbeat event

  • Header, 19 bytes

  • Content, string

This page is licensed: CC BY-SA / Gnu FDL

4-Semi-Sync Replication

Regular MariaDB replication is asynchronous. MariaDB, since MariaDB 5.5, has included semisynchronous replication semi-synchronous Binlog Event.

Event Header Changes

If the user variable @rpl_semi_sync_slave is set, 2 exta bytes are added after the status byte of a binlog network stream and before the normal binlog event header.

  • uint<1> semi-sync indicator, always 0xef

  • uint<1> semi-sync flags, either 0x00 (no ACK) or 0x01 (ACK)

Note : The packet size, as in the network protocol header, is then: event_size + 1 byte status + 2 bytes semi-sync replication.

The MariaDB server sets the user variable whenever it is starting replication. For MariaDB Connector/C , the following query must be executed before the call to mariadb_rpl_open() is made to enable semi-sync replication.

SET @rpl_semi_sync_slave=1

If the semi-sync flag is set to 0x01, the master waits for a Semi Sync ACK packet from the slave and when the Semi Sync ACK is seen, the master acknowledges the client which has issued the transaction with a standard OK_Packet or a ERR_Packet.

The master can then write the transaction to the binary log and send the next events to the slave.

Note : The master only requests Semi Sync ACKs if rpl_semi_sync_master_enabled is enabled. If it is not enabled, the semi-sync flag will always be 0x00.

Semi Sync ACK Details

This event is sent by the slave only if the semi-sync flag is set to 0x01.

  • uint<1> semi-sync indicator, always 0xef

  • uint<8> the next position of received event

  • string binlog file name.

Note: this packet sent by the slave never includes the CRC32.

Sending an ACK when the semi-sync flag is set to 0x0 will cause an error and the connection is closed.

Example of Heartbeat Event With Semi-Sync Protocol and CRC32

We can clearly see:

  • 2a 00 00 [3 bytes] packet size:

  • 06 [1] sequence

  • 00 [1] status byte = 00 => OK

  • ef 00 [2] bytes => semi sync indicator (0xef) and semi-sync flag (00)

the ef 00 2 bytes after the OK byte 00

T 127.0.0.1:23240 -> 127.0.0.1:41054 [AP]
  2a 00 00 06 00 ef 00 00    00 00 00 1b d9 27 00 00    *............'..
  27 00 00 00 79 04 00 00    00 00 6d 79 73 71 6c 2d    '...y.....mysql-
  62 69 6e 2e 30 30 30 30    33 34 ed ef e1 f0          bin.000034....

Example of XID_EVENT, With CRC32

The master sets the Semi-Sync ACK request in the XID_EVENT event:

We see the 2 semi sync bytes: ef and 01. The latter, being 1, means the slave server must send the Semi Sync ACK packet.

We also see in the binlog event header:

  • Event Type [1] = 10 XID_EVENT

  • Next Event pos [4] = 4a 05 00 00 => 1354

22 00 00 0c 00 ef 01 17  d0 37 5a 17 d0 37 5a 10  "............7Z.
d9 27 00 00 1f 00 00 00  4a 05 00 00 00 00 6f 00  .?......J.....o.
00 00 00 00 00 00 44 30  aa fc                    ......D0..

Example of Semi-Sync ACK

This is sent by the slave server after the XID_EVENT receiving.

We see:

  • the semi sync indicator [1] = 0xef, sent before anything else

  • the Next Event position [8] = 4a 05 00 00 00 00 00 00 => 1354 which is the next position of the XID_EVENT above

  • the binlog filename = mysql-bin.000034

Please note:

  • there is no terminating CRC32

  • the packet sequence now start starts from 0

19 00 00 00 ef 4a 05 00    00 00 00 00 00 6d 79 73    .....J.......mys
  71 6c 2d 62 69 6e 2e 30    30 30 30 33 34             ql-bin.000034

This page is licensed: CC BY-SA / Gnu FDL

5-Replica Registration

The replica server, when properly configured with CHANGE MASTER TO ... can start MariaDB replication with the START REPLICA command.

After authentication, some COM_QUERY packets are exchanged before sending COM_REGISTER_SLAVE and COM_BINLOG_DUMP

The following COM_QUERY packets come from MariaDB 10.X slaves using GTID

  • SELECT UNIX_TIMESTAMP()

  • SHOW VARIAB LES LIKE 'SERVER_ID'

  • SET @master_heartbeat_period= 30000001024

  • SET @master_binlog_checksum= @@global.binlog_checksum

  • SELECT @master_binlog_checksum

  • SET @mariadb_slave_capability=4

  • SELECT @@GLOBAL.gtid_domain_id GTID registration: domain ID

  • SET @slave_connect_state='0-10201-9868' GTID registration: the requested GTID

  • SET @slave_gtid_strict_mode=0 GTID registration: strict_mode

  • SET @slave_gtid_ignore_duplicates=0 GTID registration: ignore_duplicates

Then COM_REGISTER_SLAVE completes the registration.

The COM_BINLOG_DUMP marks the request of binlog events stream.

Note: If semi-sync is in use, the request for the network protocol change is sent between COM_REGISTER_SLAVE and COM_BINLOG_DUMP.

Example Using 'ngrep'

COM_REGISTER_SLAVE, Semi-Sync and COM_BINLOG_DUMP

T 127.0.0.1:42158 -> 127.0.0.1:23240 [AP]
  1a 00 00 00 15 75 27 00    00 08 53 42 73 6c 61 76    .....u'...SBslav
  65 31 00 00 c9 5a 00 00    00 00 00 00 00 00          e1...Z........  

T 127.0.0.1:23240 -> 127.0.0.1:42158 [AP]
  07 00 00 01 00 00 00 02    00 00 00                   ...........     

T 127.0.0.1:42158 -> 127.0.0.1:23240 [AP]
  33 00 00 00 03 53 48 4f    57 20 56 41 52 49 41 42    3....SHOW VARIAB
  4c 45 53 20 4c 49 4b 45    20 27 72 70 6c 5f 73 65    LES LIKE 'rpl_se
  6d 69 5f 73 79 6e 63 5f    6d 61 73 74 65 72 5f 65    mi_sync_master_e
  6e 61 62 6c 65 64 27                                  nabled'         

T 127.0.0.1:23240 -> 127.0.0.1:42158 [AP]
  01 00 00 01 02 64 00 00    02 03 64 65 66 12 69 6e    .....d....def.in
  66 6f 72 6d 61 74 69 6f    6e 5f 73 63 68 65 6d 61    formation_schema
  11 53 45 53 53 49 4f 4e    5f 56 41 52 49 41 42 4c    .SESSION_VARIABL
  45 53 11 53 45 53 53 49    4f 4e 5f 56 41 52 49 41    ES.SESSION_VARIA
  42 4c 45 53 0d 56 61 72    69 61 62 6c 65 5f 6e 61    BLES.Variable_na
  6d 65 0d 56 41 52 49 41    42 4c 45 5f 4e 41 4d 45    me.VARIABLE_NAME
  0c 08 00 40 00 00 00 fd    01 00 00 00 00 5d 00 00    ...@.........]..
  03 03 64 65 66 12 69 6e    66 6f 72 6d 61 74 69 6f    ..def.informatio
  6e 5f 73 63 68 65 6d 61    11 53 45 53 53 49 4f 4e    n_schema.SESSION
  5f 56 41 52 49 41 42 4c    45 53 11 53 45 53 53 49    _VARIABLES.SESSI
  4f 4e 5f 56 41 52 49 41    42 4c 45 53 05 56 61 6c    ON_VARIABLES.Val
  75 65 0e 56 41 52 49 41    42 4c 45 5f 56 41 4c 55    ue.VARIABLE_VALU
  45 0c 08 00 00 08 00 00    fd 01 00 00 00 00 05 00    E...............
  00 04 fe 00 00 22 00 20    00 00 05 1c 72 70 6c 5f    .....". ....rpl_
  73 65 6d 69 5f 73 79 6e    63 5f 6d 61 73 74 65 72    semi_sync_master
  5f 65 6e 61 62 6c 65 64    02 4f 4e 05 00 00 06 fe    _enabled.ON.....
  00 00 22 00                                           ..". 

T 127.0.0.1:42158 -> 127.0.0.1:23240 [AP]
  1c 00 00 00 03 53 45 54    20 40 72 70 6c 5f 73 65    .....SET @rpl_se
  6d 69 5f 73 79 6e 63 5f    73 6c 61 76 65 3d 20 31    mi_sync_slave= 1

T 127.0.0.1:23240 -> 127.0.0.1:42158 [AP]
  07 00 00 01 00 00 00 02    00 00 00                   ...........     

T 127.0.0.1:42158 -> 127.0.0.1:23240 [AP]
  1b 00 00 00 12 34 06 00    00 02 00 75 27 00 00 6d    .....4.....u'..m
  79 73 71 6c 2d 62 69 6e    2e 30 30 30 30 33 34       ysql-bin.000034

In the example we clearly see that these two COM_QUERY commands:

  • SHOW VARIABLES LIKE 'rpl_semi_sync_master_enabled'

  • SET @rpl_semi_sync_slave= 1

are sent just after COM_REGISTER_SLAVE and before COM_BINLOG_DUMP.

Complete Example with GTID Registration (Up to COM_BINLOG_DUMP Request), No Semi-Sync

T 127.0.0.1:23240 -> 127.0.0.1:42367 [AP]
  5d 00 00 00 0a 35 2e 35    2e 35 2d 31 30 2e 32 2e    ]....5.5.5-10.2.
  31 30 2d 4d 61 72 69 61    44 42 2d 6c 6f 67 00 22    10-MariaDB-log."
  00 00 00 7d 2e 6a 4f 2c    2c 36 6a 00 fe f7 08 02    ...}.jO,,6j.....
  00 bf 81 15 00 00 00 00    00 00 07 00 00 00 38 74    ..............8t
  60 64 54 59 44 28 38 24    48 7c 00 6d 79 73 71 6c    `dTYD(8$H|.mysql
  5f 6e 61 74 69 76 65 5f    70 61 73 73 77 6f 72 64    _native_password
  00                                                    .               

T 127.0.0.1:42367 -> 127.0.0.1:23240 [AP]
  a9 00 00 01 05 a2 38 80    70 03 00 40 08 00 00 00    ......8.p..@....
  00 00 00 00 00 00 00 00    00 00 00 00 00 00 00 00    ................
  00 00 00 00 6d 73 61 6e    64 62 6f 78 00 14 52 42    ....msandbox..RB
  0b e8 ae 56 ec ff ef 1f    1f 14 51 1d 4a 47 f4 32    ...V......Q.JG.2
  56 74 6d 79 73 71 6c 5f    6e 61 74 69 76 65 5f 70    Vtmysql_native_p
  61 73 73 77 6f 72 64 00    54 03 5f 6f 73 05 4c 69    assword.T._os.Li
  6e 75 78 0c 5f 63 6c 69    65 6e 74 5f 6e 61 6d 65    nux._client_name
  08 6c 69 62 6d 79 73 71    6c 04 5f 70 69 64 05 33    .libmysql._pid.3
  30 30 31 33 0f 5f 63 6c    69 65 6e 74 5f 76 65 72    0013._client_ver
  73 69 6f 6e 07 31 30 2e    32 2e 31 30 09 5f 70 6c    sion.10.2.10._pl
  61 74 66 6f 72 6d 06 78    38 36 5f 36 34             atform.x86_64   

T 127.0.0.1:23240 -> 127.0.0.1:42367 [AP]
  07 00 00 02 00 00 00 02    00 00 00                   ...........     

T 127.0.0.1:42367 -> 127.0.0.1:23240 [AP]
  18 00 00 00 03 53 45 4c    45 43 54 20 55 4e 49 58    .....SELECT UNIX
  5f 54 49 4d 45 53 54 41    4d 50 28 29                _TIMESTAMP()    

T 127.0.0.1:23240 -> 127.0.0.1:42367 [AP]
  01 00 00 01 01 26 00 00    02 03 64 65 66 00 00 00    .....&....def...
  10 55 4e 49 58 5f 54 49    4d 45 53 54 41 4d 50 28    .UNIX_TIMESTAMP(
  29 00 0c 3f 00 11 00 00    00 08 80 00 00 00 00 05    )..?............
  00 00 03 fe 00 00 02 00    0b 00 00 04 0a 31 35 31    .............151
  33 36 38 34 33 38 36 05    00 00 05 fe 00 00 02 00    3684386.........

T 127.0.0.1:42367 -> 127.0.0.1:23240 [AP]
  20 00 00 00 03 53 48 4f    57 20 56 41 52 49 41 42     ....SHOW VARIAB
  4c 45 53 20 4c 49 4b 45    20 27 53 45 52 56 45 52    LES LIKE 'SERVER
  5f 49 44 27                                           _ID'            

T 127.0.0.1:23240 -> 127.0.0.1:42367 [AP]
  01 00 00 01 02 64 00 00    02 03 64 65 66 12 69 6e    .....d....def.in
  66 6f 72 6d 61 74 69 6f    6e 5f 73 63 68 65 6d 61    formation_schema
  11 53 45 53 53 49 4f 4e    5f 56 41 52 49 41 42 4c    .SESSION_VARIABL
  45 53 11 53 45 53 53 49    4f 4e 5f 56 41 52 49 41    ES.SESSION_VARIA
  42 4c 45 53 0d 56 61 72    69 61 62 6c 65 5f 6e 61    BLES.Variable_na
  6d 65 0d 56 41 52 49 41    42 4c 45 5f 4e 41 4d 45    me.VARIABLE_NAME
  0c 08 00 40 00 00 00 fd    01 00 00 00 00 5d 00 00    ...@.........]..
  03 03 64 65 66 12 69 6e    66 6f 72 6d 61 74 69 6f    ..def.informatio
  6e 5f 73 63 68 65 6d 61    11 53 45 53 53 49 4f 4e    n_schema.SESSION
  5f 56 41 52 49 41 42 4c    45 53 11 53 45 53 53 49    _VARIABLES.SESSI
  4f 4e 5f 56 41 52 49 41    42 4c 45 53 05 56 61 6c    ON_VARIABLES.Val
  75 65 0e 56 41 52 49 41    42 4c 45 5f 56 41 4c 55    ue.VARIABLE_VALU
  45 0c 08 00 00 08 00 00    fd 01 00 00 00 00 05 00    E...............
  00 04 fe 00 00 22 00 10    00 00 05 09 73 65 72 76    ....."......serv
  65 72 5f 69 64 05 31 30    32 30 31 05 00 00 06 fe    er_id.10201.....
  00 00 22 00                                           ..".            

T 127.0.0.1:42367 -> 127.0.0.1:23240 [AP]
  2a 00 00 00 03 53 45 54    20 40 6d 61 73 74 65 72    *....SET @master
  5f 68 65 61 72 74 62 65    61 74 5f 70 65 72 69 6f    _heartbeat_perio
  64 3d 20 33 30 30 30 30    30 30 31 30 32 34          d= 30000001024  

T 127.0.0.1:23240 -> 127.0.0.1:42367 [AP]
  07 00 00 01 00 00 00 02    00 00 00                   ...........     

T 127.0.0.1:42367 -> 127.0.0.1:23240 [AP]
  36 00 00 00 03 53 45 54    20 40 6d 61 73 74 65 72    6....SET @master
  5f 62 69 6e 6c 6f 67 5f    63 68 65 63 6b 73 75 6d    _binlog_checksum
  3d 20 40 40 67 6c 6f 62    61 6c 2e 62 69 6e 6c 6f    = @@global.binlo
  67 5f 63 68 65 63 6b 73    75 6d                      g_checksum      

T 127.0.0.1:23240 -> 127.0.0.1:42367 [AP]
  07 00 00 01 00 00 00 02    00 00 00                   ...........     

T 127.0.0.1:42367 -> 127.0.0.1:23240 [AP]
  1f 00 00 00 03 53 45 4c    45 43 54 20 40 6d 61 73    .....SELECT @mas
  74 65 72 5f 62 69 6e 6c    6f 67 5f 63 68 65 63 6b    ter_binlog_check
  73 75 6d                                              sum             

T 127.0.0.1:23240 -> 127.0.0.1:42367 [AP]
  01 00 00 01 01 2d 00 00    02 03 64 65 66 00 00 00    .....-....def...
  17 40 6d 61 73 74 65 72    5f 62 69 6e 6c 6f 67 5f    .@master_binlog_
  63 68 65 63 6b 73 75 6d    00 0c 08 00 ff ff ff 00    checksum........
  fa 00 00 27 00 00 05 00    00 03 fe 00 00 02 00 06    ...'............
  00 00 04 05 43 52 43 33    32 05 00 00 05 fe 00 00    ....CRC32.......
  02 00                                                 ..              

T 127.0.0.1:42367 -> 127.0.0.1:23240 [AP]
  20 00 00 00 03 53 45 54    20 40 6d 61 72 69 61 64     ....SET @mariad
  62 5f 73 6c 61 76 65 5f    63 61 70 61 62 69 6c 69    b_slave_capabili
  74 79 3d 34                                           ty=4            

T 127.0.0.1:23240 -> 127.0.0.1:42367 [AP]
  07 00 00 01 00 00 00 02    00 00 00                   ...........     

T 127.0.0.1:42367 -> 127.0.0.1:23240 [AP]
  1f 00 00 00 03 53 45 4c    45 43 54 20 40 40 47 4c    .....SELECT @@GL
  4f 42 41 4c 2e 67 74 69    64 5f 64 6f 6d 61 69 6e    OBAL.gtid_domain
  5f 69 64                                              _id             

T 127.0.0.1:23240 -> 127.0.0.1:42367 [AP]
  01 00 00 01 01 2d 00 00    02 03 64 65 66 00 00 00    .....-....def...
  17 40 40 47 4c 4f 42 41    4c 2e 67 74 69 64 5f 64    .@@GLOBAL.gtid_d
  6f 6d 61 69 6e 5f 69 64    00 0c 3f 00 15 00 00 00    omain_id..?.....
  08 a0 00 00 00 00 05 00    00 03 fe 00 00 02 00 02    ................
  00 00 04 01 30 05 00 00    05 fe 00 00 02 00          ....0.........  

T 127.0.0.1:42367 -> 127.0.0.1:23240 [AP]
  28 00 00 00 03 53 45 54    20 40 73 6c 61 76 65 5f    (....SET @slave_
  63 6f 6e 6e 65 63 74 5f    73 74 61 74 65 3d 27 30    connect_state='0
  2d 31 30 32 30 31 2d 39    38 36 38 27                -10201-9868'    

T 127.0.0.1:23240 -> 127.0.0.1:42367 [AP]
  07 00 00 01 00 00 00 02    00 00 00                   ...........     

T 127.0.0.1:42367 -> 127.0.0.1:23240 [AP]
  1e 00 00 00 03 53 45 54    20 40 73 6c 61 76 65 5f    .....SET @slave_
  67 74 69 64 5f 73 74 72    69 63 74 5f 6d 6f 64 65    gtid_strict_mode
  3d 30                                                 =0              

T 127.0.0.1:23240 -> 127.0.0.1:42367 [AP]
  07 00 00 01 00 00 00 02    00 00 00                   ...........     

T 127.0.0.1:42367 -> 127.0.0.1:23240 [AP]
  24 00 00 00 03 53 45 54    20 40 73 6c 61 76 65 5f    $....SET @slave_
  67 74 69 64 5f 69 67 6e    6f 72 65 5f 64 75 70 6c    gtid_ignore_dupl
  69 63 61 74 65 73 3d 30                               icates=0        

T 127.0.0.1:23240 -> 127.0.0.1:42367 [AP]
  07 00 00 01 00 00 00 02    00 00 00                   ...........     

T 127.0.0.1:42367 -> 127.0.0.1:23240 [AP]
  1b 00 00 00 15 75 27 00    00 09 73 6c 61 76 65 5f    .....u'...slave_
  6e 5f 31 00 00 c9 5a 00    00 00 00 00 00 00 00       n_1...Z........ 

T 127.0.0.1:23240 -> 127.0.0.1:42367 [AP]
  07 00 00 01 00 00 00 02    00 00 00                   ...........     

T 127.0.0.1:42367 -> 127.0.0.1:23240 [AP]
  1b 00 00 00 12 34 06 00    00 02 00 75 27 00 00 6d    .....4.....u'..m
  79 73 71 6c 2d 62 69 6e    2e 30 30 30 30 33 34       ysql-bin.000034

Events Transmission After COM_BINLOG_DUMP.

The MariaDB 10.x Master always sends, after the COM_BINLOG_DUMP:

  • FAKE_ROTATE_EVENT

  • FORMAT_DESCRIPTION_EVENT: Next Pos in the header is set to 0 if not requesting binlog file form the beginning and GTID is not in use otherwise Next Pos is related to next event after FDE

  • FAKE_GTID_LIST_EVENT with latest GTID information.

After those first events, the master sends events related to changes in database to the connected replica binlog. The replica is just waiting for new events from master.

Complete Example of Event Transmission With CRC32

T 127.0.0.1:23240 -> 127.0.0.1:42219 [AP]
  30 00 00 01 00 00 00 00    00 04 d9 27 00 00 2f 00    0..........'../.
  00 00 00 00 00 00 20 00    04 00 00 00 00 00 00 00    ...... .........
  6d 79 73 71 6c 2d 62 69    6e 2e 30 30 30 30 33 34    mysql-bin.000034
  d5 3f ea d7 fd 00 00 02    00 fb cc 37 5a 0f d9 27    .?.........7Z..'
  00 00 fc 00 00 00 00 01    00 00 00 00 04 00 31 30    ..............10
  2e 32 2e 31 30 2d 4d 61    72 69 61 44 42 2d 6c 6f    .2.10-MariaDB-lo
  67 00 00 00 00 00 00 00    00 00 00 00 00 00 00 00    g...............
  00 00 00 00 00 00 00 00    00 00 00 00 00 00 00 00    ................
  00 00 00 00 13 38 0d 00    08 00 12 00 04 04 04 04    .....8..........
  12 00 00 e4 00 04 1a 08    00 00 00 08 08 08 02 00    ................
  00 00 0a 0a 0a 00 00 00    00 00 00 00 00 00 00 00    ................
  00 00 00 00 00 00 00 00    00 00 00 00 00 00 00 00    ................
  00 00 00 00 00 00 00 00    00 00 00 00 00 00 00 00    ................
  00 00 00 00 00 00 00 00    00 00 00 00 00 00 00 00    ................
  00 00 00 00 00 00 00 00    00 00 00 00 00 00 00 00    ................
  00 00 00 00 00 00 00 00    00 00 00 00 00 00 00 00    ................
  00 00 00 00 00 00 00 00    00 00 00 00 00 00 00 00    ................
  00 00 00 00 00 00 00 00    00 00 00 00 00 00 00 00    ................
  00 00 00 00 00 04 13 04    00 0d 08 08 08 0a 0a 0a    ................
  01 17 0b 12 63 3c 00 00    03 00 fb cc 37 5a a3 d9    ....c<......7Z..
  27 00 00 3b 00 00 00 3b    01 00 00 00 00 02 00 00    '..;...;........
  00 00 00 00 00 01 00 00    00 1e 00 00 00 00 00 00    ................
  00 00 00 00 00 d9 27 00    00 86 26 00 00 00 00 00    ......'...&.....
  00 b6 33 8a 22 2c 00 00    04 00 fb cc 37 5a a1 d9    ..3.",......7Z..
  27 00 00 2b 00 00 00 66    01 00 00 00 00 10 00 00    '..+...f........
  00 6d 79 73 71 6c 2d 62    69 6e 2e 30 30 30 30 33    .mysql-bin.00003
  34 16 1f fe 3f 2c 00 00    05 00 00 00 00 00 a3 d9    4...?,..........
  27 00 00 2b 00 00 00 34    06 00 00 20 00 01 00 00    '..+...4... ....
  00 00 00 00 00 d9 27 00    00 8c 26 00 00 00 00 00    ......'...&.....
  00 4a 01 94 22 2b 00 00    06 00 94 fd 38 5a a2 d9    .J.."+......8Z..
  27 00 00 2a 00 00 00 5e    06 00 00 08 00 8d 26 00    '..*...^......&.
  00 00 00 00 00 00 00 00    00 29 00 00 00 00 00 00    .........)......
  22 87 c0 61 4c 00 00 07    00 94 fd 38 5a 02 d9 27    "..aL......8Z..'
  00 00 4b 00 00 00 a9 06    00 00 00 00 21 00 00 00    ..K.........!...
  00 00 00 00 00 00 00 1a    00 00 00 00 00 00 01 00    ................
  00 20 54 00 00 00 00 06    03 73 74 64 04 08 00 08    . T......std....
  00 08 00 00 66 6c 75 73    68 20 74 61 62 6c 65 73    ....flush tables
  6e c8 89 60                                           n..`

We can see:

  1. FAKE_ROTATE_EVENT packet: 30 00 00 01 ... d5 3f ea d7

  2. FORMAT_DESCRIPTION_EVENT packet: fd 00 00 02 00 ... 17 0b 12 63 FDE size is fc 00 00 00 (252) Next Pos in FDE is 00 01 00 00 = >256 = 4 + FDE size (252)

  3. FAKE GTID_LIST_EVENT packet : 3c 00 00 03 00 ... b6 33 8a 22

  4. BINLOG_CHECKPOINT EVENT packet: 2c 00 00 04 ... 16 1f fe 3f

  5. GTID_LIST_EVENT packet: 2c 00 00 05 ... 4a 01 94 22

  6. GTID_EVENT packet: 2b 00 00 06 ... 22 87 c0 61

  7. QUERY_EVENT packet: 4c 00 00 07 ... 6e c8 89 60

This page is licensed: CC BY-SA / Gnu FDL

ANNOTATE_ROWS_EVENT

ANNOTATE_ROWS_EVENT events accompany row events and describe the query which caused the row event.

You can enable this with --binlog-annotate-row-events (default on from MariaDB 10.2.4).

In the binary log, each Annotate_rows event precedes the corresponding Table map event.

For additional information refer to the annotate_rows_log_event documentation.

Note: the master server sends ANNOTATE_ROWS_EVENT events only if the Slave server connects with the BINLOG_SEND_ANNOTATE_ROWS_EVENT flag (value is 2) in the COM_BINLOG_DUMP Slave Registration phase.

Header

  • Event Type is 160 (0xa0)

Fields

  • string The SQL statement (not null-terminated)

Complete Example with CRC32

ee b7 15 5a a0 01 00 00  00 36 00 00 00 80 0b 00  ...Z.....6......
00 00 00 69 6e 73 65 72  74 20 69 6e 74 6f 20 74  ...insert into t
65 73 74 2e 74 34 20 76  61 6c 75 65 73 28 31 30  est.t4 values(10
30 29 6d 4c 42 33                                 0)mLB3

This page is licensed: CC BY-SA / Gnu FDL

BEGIN_LOAD_QUERY_EVENT

This event is written into the binary log file for LOAD DATA INFILE events if the server variable binlog_mode was set to "STATEMENT".

Header

  • Event Type = 0x11

Fields

Fixed data part:

  • uint<4> The ID of the file

Variable data part:

  • byte Null terminated data block.

Example

TODO: Add an example

This page is licensed: CC BY-SA / Gnu FDL

BINLOG_CHECKPOINT_EVENT

Binlog Checkpoint Event, Event Type is 161 (0xa1) This event specifies a binlog file such that XA crash recovery can start from that file.

Note: there can be more than one in a binlog file.

Header

  • Event type is 161 (0xa1)

Fields

  • uint<4> Log filename length

  • string Log filename

Example Without CRC32

12 ad 26 5a a1 84 27 00  00 27 00 00 00 47 01 00  ..&Z..'..'...G..
00 00 00 10 00 00 00 6d  79 73 71 6c 2d 62 69 6e  .......mysql-bin
2e 30 30 30 30 36 32                              .000062

Header, 19 Bytes

  • Event Timestamp = 12 ad 26 5a

  • Event Type = a1 => 161

  • Server_id = 84 27 00 00 => 00 00 27 84 = 10116

  • Event Size = 27 00 00 00 => 00 00 00 27 = 39

  • Next Pos = 47 01 00 00 => 00 00 01 47 = 327

  • Flags = 00 => 0

Content, Variable Size

  • filename length = 10 00 00 00 = >00 00 00 10 => 16

  • filename = mysql-bin.000062

This page is licensed: CC BY-SA / Gnu FDL

COM_BINLOG_DUMP

This is a command the slave sends to the master after COM_REGISTER_SLAVE.

The master server sends the binlog events from the requested file and position, or if GTID registration is in use, from the GTID value set in the earlier registration phase.

The payload is:

  • uint<1> command (COM_BINLOG_DUMP = 0x12)

  • uint<4> The requested binlog position

  • uint<2> Flags

  • uint<4> Slave server_id

  • string The requested binlog file name

Note

  • Flags, usually set to 0. It can be set to BINLOG_SEND_ANNOTATE_ROWS_EVENT (0x02) if the slave server wants to receive the MariaDB 10 ANNOTATE_ROWS events. It can also be set to BINLOG_DUMP_NON_BLOCK (1): in the case the slave is receiving an EOF packet after last event sent by the master.

  • Requested binlog position can be 4 when registering to master server for the very first time or when requesting events from a particular binlog file from the beginning of it.

  • The requested binlog file can empty when registering for the very first time if master log file is unknown or with GTID registration (not required).

  • After sending events to the replica the server kills connection.

When replication resumes or it is restarted (STOP SLAVE; START SLAVE) the slave server always sends the latest binlog file name and position even if GTID registration is in place.

Example of COM_BINLOG_DUMP

1b 00 00 00 12 34 06 00    00 02 00 75 27 00 00 6d    .....4.....u'..m
  79 73 71 6c 2d 62 69 6e    2e 30 30 30 30 33 34       ysql-bin.000034

After 4 bytes network protocol header we can see:

  • command [1] = 12

  • requested binlog position [4] = 34 06 00 00 => 00 00 06 34 = 1588

  • flags [2] = 02 00 => 2 = BINLOG_SEND_ANNOTATE_ROWS_EVENT

  • binlog file[n] = mysql-bin.000034

This page is licensed: CC BY-SA / Gnu FDL

COM_REGISTER_SLAVE

This command is sent by the slave server in order to start MariaDB replication and should be sent before requesting binlog events with COM_BINLOG_DUMP.

The payload is:

  • uint<1> command (COM_REGISTER_SLAVE = 0x15)

  • uint<4> Slave server_id

  • uint<1> Slave hostname length

  • string Hostname

  • uint<1> Slave username len

  • string Username

  • uint<1> Slave password len

  • string Slave password

  • uint<2> Slave connection port

  • uint<4> Replication rank

  • uint<4> Master server id

Note:

  • Slave hostname, slave user, slave password and slave port are usually not set. Some slave replication parameters can be used for such settings (report_host, report_port etc).

  • Replication rank is not set.

  • Master server id is not set as well.

Example of COM_REGISTER_SLAVE

The slave server is configured with:

  • server-id=10101

  • report-host=slave_n_1

  • report-port=23241

1b 00 00 00 15 75 27 00    00 09 73 6c 61 76 65 5f    .....u'...slave_
 6e 5f 31 00 00 c9 5a 00    00 00 00 00 00 00 00       n_1...Z.......

We can see from the example:

  • server_id [4] = 75 27 00 00 => 10101

  • hostname_len [1] = 09

  • hostname[n] = slave_n_1 (9 bytes)

  • username len [1] = 0 (not set)

  • password len [1] = 0 (not set)

  • slave port [2] = c9 5a => 23241

  • rank [4] = 0

  • master server id = 0

This page is licensed: CC BY-SA / Gnu FDL

EXECUTE_LOAD_QUERY_EVENT

This event is written into the binary log file for LOAD DATA INFILE events. The event format is similar to a QUERY_EVENT except that it has extra static fields.

Header

  • Event Type = 0x12

Fields

Fixed data part:

  • uint<4> The ID of the thread that issued this statement on the master.

  • uint<4> The time in seconds that the statement took to execute.

  • uint<1> The length of the name of the database which was the default database when the statement was executed. This name appears later, in the variable data part. It is necessary for statements such as INSERT INTO t VALUES(1) that don't specify the database and rely on the default database previously selected by USE.

  • uint<2> The error code resulting from execution of the statement on the master.

  • uint<2> The length of the status variable block.

  • uint<4> The ID of the loaded file

  • uint<4> Offset from the start of the statement to the beginning of the filename

  • uint<4> Offset from the start of the statement to the end of the filename

  • uint<1> How LOAD DATA INFILE handles duplicates (0x0: error, 0x1: ignore, 0x2: replace).

Variable data part:

  • byte Zero or more status variables. Each status variable consists of one byte code identifying the variable stored, followed by the value of the variable. The format of the value is variable-specific. The number of bytes 'n' is the length of the status variable block (read in fixed data part)

  • string The default database name (null-terminated).

  • string The SQL statement. By subtraction the size of the statement can be known.

Example

TODO: Add an example

This page is licensed: CC BY-SA / Gnu FDL

Fake GTID_LIST event

This event is sent by master server to the registering slave and it's sent only once, afterFormat Description Event

Note: the fake GTID_LIST event is not written in the binlog file. It's created by the master and sent to new connected slave before any "real" binlog event.

Header

  • Event type is set to 163 (0xa3)

  • Time stamp set to 0

  • NextPos tells which is the binlog position of next event

  • Flags are set to ARTIFICIAL (0x20)

Content

The content is the same as the "real" GTID_LIST.

  • of GTIDs

  • domain_id

  • server_id

  • sequence

  • ...

This page is licensed: CC BY-SA / Gnu FDL

Fake ROTATE_EVENT

When a slave server connects to a MariaDB 10 master server, the first binlog event sent is Fake ROTATE_EVENT. This event is similar to ROTATE_EVENT but it's artificial and its purpose is to tell the slave server which is the binlog file name of the master. This matters of course when the slave connects with the GTID option (no filename is given) or when using file and pos with empty file name (usually file='' and pos = 4).

The Event Type is set ROTATE_EVENT (0x4)

Note: the fake ROTATE_EVENT event is not written in the binlog file. It's created by the master and sent to new connected slave before FORMAT_DESCRIPTION_EVENT

Header

  • Timestamp set to 0

  • Event Tye is ROTATE_EVENT

  • Next Pos is set to 0

  • Flags are set to LOG_ARTIFICIAL_F (0x20)

Content

The content is the same as ROTATE_EVENT.

  • pos = the requested pos from slave, usually 4

  • filename = the master binlog filename

If it is the first fake rotate event and global server variable @@binlog_checksum was set to CRC32:

  • crc32_checksum (4 Bytes)

This page is licensed: CC BY-SA / Gnu FDL

FORMAT_DESCRIPTION_EVENT

This is a descriptor event that is written to the beginning of a binary log file, at position 4 (after the 4 magic number bytes)

The whole event written to disk is byte<19> event header + data fields

Header

  • The Event Type is 15 (0x0f)

Fields

  • uint<2> The binary log format version. This is 4 in MariaDB 10 and up.

  • string<50> The MariaDB server version (example: 10.2.1-debug-log), padded with 0x00 bytes on the right.

  • uint<4> Timestamp in seconds when this event was created (this is the moment when the binary log was created). This value is redundant; the same value occurs in the timestamp header field.

  • uint<1> The header length. This length - 19 gives the size of the extra headers field at the end of the header for other events.

  • byte Variable-sized. An array that indicates the post-header lengths for all event types. There is one byte per event type that the server knows about. The value 'n' comes from the following formula:

n = event_size - header length - offset (2 + 50 + 4 + 1) - checksum_algo - checksum
  • uint<1> Checksum Algorithm Type

  • uint<4> CRC32 4 bytes (value matters only if checksum algo is CRC32)

Example FDE of MariaDB 10.2.10 With CRC32

4d af 15 5a 0f 01 00 00  00 fc 00 00 00 00 01 00  M..Z............
00 00 00 04 00 31 30 2e  32 2e 31 30 2d 4d 61 72  .....10.2.10-Mar
69 61 44 42 2d 6c 6f 67  00 00 00 00 00 00 00 00  iaDB-log.......
00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ................
00 00 00 00 00 00 00 00  00 00 00 4d af 15 5a 13  ...........M..Z.
00 12 00 04 04 04 04 12  00 00 e4 00 04 1a 08 00  ................
00 00 08 08 08 02 00 00  00 0a 0a 0a 00 00 00 00  ................
00 00 00 00 00 00 00 00  00 00 00 00 04 13 04 00  ................
00 00 00 00 00 00 00 00  00 00 00 00 04 13 04 00  ................
00 00 00 00 00 00 00 00  00 00 00 00 04 13 04 00  ................
00 00 00 00 00 00 00 00  00 00 00 00 04 13 04 00  ................
00 00 00 00 00 00 00 00  00 00 00 00 04 13 04 00  ................
00 00 00 00 00 00 00 00  00 00 00 00 04 13 04 00  ................
00 00 00 00 00 00 00 00  00 00 00 00 04 13 04 00  ................
00 00 00 00 00 00 00 00  00 00 00 00 04 13 04 00  ................
0d 08 08 08 0a 0a 0a 01  d6 ce 13 e2              ............

Header, 19 Bytes

  • timestamp => 4d af 15 5a

  • type = 0f => 15

  • server_id = 1

  • Event Size = fc => 252

  • Next Pos = 00 01 00 00 => 00 00 01 00 => 256

  • Flags = 00 => 0

Content, Variable Size Depending on MariaDB Versions

  • format version = 04 00 => 4

  • server's version = 10.2.10-MariaDB-log .... [50 bytes]

  • create time = 4d af 15 5a

  • header_length = 13 => 19

  • event_types array[252 - 19 - (2 + 50 + 4 +1) - 1 - 4] = 171 supported events

  • checksum_algo = 01 => 1 (CRC32)

  • CRC32 bytes = d6 ce 13 e2

Example FDE of MariaDB 10.1.16 With CRC32

12 ad 26 5a 0f 84 27 00  00 f5 00 00 00 f9 00 00  ..&Z..'.........
00 01 00 04 00 31 30 2e  31 2e 31 36 2d 4d 61 72  .....10.1.16-Mar
69 61 44 42 00 6c 6f 67  00 00 00 00 00 00 00 00  iaDB.log........
00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ................
00 00 00 00 00 00 00 12  ad 26 5a 13 38 0d 00 08  .........&Z.8...
00 12 00 04 04 04 04 12  00 00 dd 00 04 1a 08 00  ................
00 00 08 08 08 02 00 00  00 0a 0a 0a 00 00 00 00  ................
00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ................
00 00 00 00 00 00 00 00  00 00 00 00 04 13 04 00  ................
00 2b 91 c2 91

Header, 19 Bytes

  • timestamp => 12 ad 26 5aa

  • type = 0f => 15

  • server_id = 84 27 00 00 => 00 00 27 84 => 10116

  • Event Size = fc => 245

  • Next Pos = f9 00 00 00 => 00 00 00 f9 => 249

  • Flags = 00 => 0

Content, Variable Size Depending on MariaDB Versions

  • format version = 04 00 => 4

  • server's version = 10.1.16-MariaDB.log .... [50 bytes]

  • create time = 12 ad 26 5a

  • header_length = 13 => 19

  • event_types array[245 - 19 - (2 + 50 + 4 +1) - 1 - 4] = 164 supported events

  • checksum_algo = 0 => 0 (NONE)

  • CRC32 bytes = 2b 91 c2 91 (useless)

This page is licensed: CC BY-SA / Gnu FDL

GTID_EVENT

For global transaction ID, used to start a new transaction event group, instead of the old BEGIN query event, and also to mark stand-alone (ddl).

GTID_EVENT, event type is 162 (0xa2)

Event Header

  • Type[1] = 0xa2

  • Flags[2] = 08 00 => LOG_EVENT_SUPPRESS_USE_F

Fields

  • uint<8> GTID sequence

  • uint<4> Replication Domain ID

  • uint<1> Flags

if flag & FL_GROUP_COMMIT_ID

  • uint<8> commit_id

else if flag & (FL_PREPARED_XA | FL_COMPLETED_XA)

  • uint<4> format_id

  • uint<1> gtid_length

  • uint<1> bqual_length

  • bytexid, where n is sum of gtrid and bqual lengths

else

  • uint<6> 0

Flags

FL_STANDALONE

1

Set when there is no terminating COMMIT event.

FL_GROUP_COMMIT_ID

2

Set when event group is part of a group commit on the master. Groups with same commit_id are part of the same group commit.

FL_TRANSACTIONAL

4

Set for an event group that can be safely rolled back (no MyISAM, eg.).

FL_ALLOW_PARALLEL

8

Reflects the (negation of the) value of @@SESSION.skip_parallel_replication at the time of commit.

FL_WAITED

16

Set if a row lock wait (or other wait) is detected during the execution of the transaction.

FL_DDL

32

Set for event group containing DDL.

FL_PREPARED_XA

64

Set for XA transaction.

FL_COMPLETED_XA

128

XA transaction completed (committed or rolled back)

Transaction Example from mysqlbinlog Utility

BEGIN
#171205 18:22:52 server id 10124  end_log_pos 652 CRC32 0x23c8d337 	GTID 0-10124-9884 trans
TBALE_MAP
#171205 18:22:52 server id 10124  end_log_pos 752 CRC32 0x52601513 	Table_map: `test`.`t4` mapped to number 92
WRITE
#171205 18:22:52 server id 10124  end_log_pos 790 CRC32 0x8869c123 	Write_rows: table id 92 flags: STMT_END_F
COMMIT
#171205 18:22:52 server id 10124  end_log_pos 821 CRC32 0x15517636 	Xid = 42004

Standalone Event DDL (FLUSH TABLES) from mysqlbinlog Utility

#171205 17:44:27 server id 10124  end_log_pos 535 CRC32 0x309a668e 	GTID 0-10124-9883 ddl
#171205 17:44:27 server id 10124  end_log_pos 610 CRC32 0xda151470 	Query	thread_id=819	...

Example GTID_EVENT with DDL and CRC32

eb cc 26 5a a2 8c 27 00  00 2a 00 00 00 17 02 00  ...&Z..'..*.....
00 08 00 9b 26 00 00 00  00 00 00 00 00 00 00 29  ...&..........).
00 00 00 00 00 00 8e 66  9a 30                    ......f.0.

Content

  • GTID seq[8] = 9b 26 00 00 00 00 00 00 => 9883

  • domain id[4] = 00 00 00 00 00 => 0

  • flags[1] = 29 => 41 (FL_DDL =32 + FL_ALLOW_PARALLEL=8 + FL_STANDALONE=1)

  • commit_id[6] = 00 00 00 00 00 00 = 0

  • CRC32[4] = 8e 66 9a 30

Example GTID_EVENT With a Transaction and CRC32

ec d5 26 5a a2 8c 27 00  00 2a 00 00 00 8c 02 00 ..&Z..'..*......
00 08 00 9c 26 00 00 00  00 00 00 00 00 00 00 0c ....&...........
00 00 00 00 00 00 37 d3  c8 23                   ......7..#

Content

  • GTID seq[8] = 9c 26 00 00 00 00 00 00 => 9884

  • domain id[4] = 00 00 00 00 00 => 0

  • flags[1] = 0c => 12 (FL_ALLOW_PARALLEL=8 + FL_TRANSACTIONAL=4)

  • commit_id[6] = 00 00 00 00 00 00 = 0

  • CRC32[4] = 37 d3 c8 23

This page is licensed: CC BY-SA / Gnu FDL

GTID_LIST_EVENT

Logged in every binlog to record the current replication state. Consists of the last GTID seen for each replication domain.

The Global Transaction ID, GTID for short, consists of three components:

  • replication domain ID

  • master server ID

  • sequence ID

It's represented as three numbers separated with dashes '-' For example:1-1222-1011

It's usually written after the Format Description Event, if binary log encryption is enabled it is written after the Start Encryption Event.

Note: In case of encrypted binlog files (encrypt_binlog is set to ON), this event is written just after the START_ENCRYPTION_EVENT

Header

  • Event type is 163 (0xa3)

Fields

  • uint<4> Number of GTIDs

  • for (i=0; i < gtid_count; i++)

    • uint<4> Replication Domain ID

    • uint<4> Server_ID

    • uint<8> GTID sequence

The minimum content size for 1 GTID is: 4 + (4 + 4 + 8) * 1 = 20 bytes

Example With 1 GTID, With CRC32

From the mysqlbinlog utility:

170824 9:52:04 server id 10124 end_log_pos 292 CRC32 0xb6d8f0a8 Gtid list [0-10124-3584]

a4 85 9e 59 a3 8c 27 00  00 2b 00 00 00 24 01 00  ...Y..'..+...$..
00 00 00 01 00 00 00 00  00 00 00 8c 27 00 00 00  ............'...
0e 00 00 00 00 00 00 a8  f0 d8 b6                 ..........

Header, 19 Bytes

  • Event Time = a4 85 9e 59 ===> 2017-08-24 9:52:04

  • Event Type = a3 => 163

  • Server_id = 8c 27 00 00 => 00 00 27 8c => 10124

  • Event Size = 2b => 43 (header[19] + 1 GTID(20 bytes) + CRC32[4]

  • Next Pos = 24 01 00 00 => 00 00 01 24 => 292

  • Flags = 00 => 0

Content, Variable Size, is (4 + (4 + 4 + 8 ) * n_GTIDs) Bytes

The content example with one GTID is 20 bytes + 4 bytes CRC32:

  • Number of GTIDs[4] = 01 00 00 00 => 1

  • GTID[0] replication_domain[4] = 00 00 00 00 => 0

  • GTID[0] Server_id[4] = 8c 27 00 00 => 00 00 27 8c => 10124

  • GTID[0] Sequence[8] = 00 0e 00 00 00 00 00 00 ===> 3584

  • crc32[4] = a8 f0 d8 b6 => b6 d8 f0 a8 => 0xb6d8f0a8

This page is licensed: CC BY-SA / Gnu FDL

HEARTBEAT_LOG_EVENT

This event does not appear in the binary log. It's only sent over the network by a master to a slave server to let it know that the master is still alive, and is only sent when the master has no binlog events to send to slave servers.

This event is never written into the binary log file

Header

  • Timestamp is set to 0

  • Next position is set to last pos

  • Type is set to HEARTBEAT_EVENT (0x1b)

Fields

  • string The current master binary log name

Example of Transmission (Without CRC32)

T 127.0.0.1:8808 -> 127.0.0.1:57157 [AP]
  23 00 00 04 00 00 00 00    00 1b 67 2b 00 00 22 00    '.........g+..&.
  00 00 ed 01 00 00 20 00    66 6f 6f 2d 62 69 6e 2e    ...... .log-bin.
  31 30 30 30 31 33 39                                  1000139

Network Replication protocol, 5 bytes

  • packet size [3] = 23 00 00 => 00 00 23 => 35 (ok byte + event size)

  • pkt sequence [1] = 04

  • OK indicator [1] = 0 (OK)

Heartbeat event

Header

  • Timestamp [4] = 00 00 00 00 => 0

  • Event Type [1] = 1b => 27

  • Server_id [4] = 67 2b 00 00 => 00 00 2b 67 => 111111

  • Event Size [4] = 22 00 00 00 => 00 00 00 26 => 34 (header + data)

  • Next_pos [4] = ed 01 00 00 => 00 00 01 ed => 493

  • Flags [2] == 20 00 => 00 20 = > 32

Content, string

  • log-bin.1000139

This page is licensed: CC BY-SA / Gnu FDL

INTVAR_EVENT

A INTVAR_EVENT is written every time a statement uses an auto increment column or LAST_INSERT_ID() function.

Header

  • Event Type is 5 (0x05)

Fields

  • uint<1> Type

  • uint<8> Value

Type

0x00

invalid value

0x01

LAST_INSERT_ID

0x02

Insert id (auto_increment)

Example From mysqlbinlog Utility, CRC32

# at 738
#180610 11:20:56 server id 1  end_log_pos 770 CRC32 0xf5a23f2d 	Intvar
SET LAST_INSERT_ID=1/*!*/;

Example Event As It's Written In The Binlog File

78 ed 1c 5b 05 01 00 00 00 20 00       x..[..... .
00 00 02 03 00 00 00 00 01 01 00 00 00 00 00 00  ................
00 2d 3f a2 f5                                   .-?..

This page is licensed: CC BY-SA / Gnu FDL

QUERY_EVENT

This event is written into the binary log file for:

  • STATEMENT based replication (updating statements)

  • DDLs

  • COMMIT related to non transactional engines (MyISAM, BLACKHOLE etc)

Header

  • Event Type = 0x02

Fields

Fixed data part:

  • uint<4> The ID of the thread that issued this statement on the master.

  • uint<4> The time in seconds that the statement took to execute.

  • uint<1> The length of the name of the database which was the default database when the statement was executed. This name appears later, in the variable data part. It is necessary for statements such as INSERT INTO t VALUES(1) that don't specify the database and rely on the default database previously selected by USE.

  • uint<2> The error code resulting from execution of the statement on the master.

  • uint<2> The length of the status variable block.

Variable data part:

  • byte Zero or more status variables. Each status variable consists of one byte code identifying the variable stored, followed by the value of the variable. The format of the value is variable-specific. The number of bytes 'n' is the length of the status variable block (read in fixed data part)

  • string The default database name (null-terminated).

  • string The SQL statement. By subtraction the size of the statement can be known.

Status variables

Q_FLAGS2_CODE (0x00):

  • uint<4> bitmask

Value

Option

0x00004000

OPTION_AUTO_IS_NULL

0x00080000

OPTION_NOT_AUTOCOMMIT

0x04000000

OPTION_NO_FOREIGN_KEY_CHECKS

0x08000000

OPTION_RELAXED_UNIQUE_CHECKS

Q_SQL_MODE_CODE (0x01):

  • uint<8> 8-byte bitmask

Value

SQL_MODE

0x00000001

MODE_REAL_AS_FLOAT

0x00000002

MODE_PIPES_AS_CONCAT

0x00000004

MODE_ANSI_QUOTES

0x00000008

MODE_IGNORE_SPACE

0x00000010

MODE_NOT_USED

0x00000020

MODE_ONLY_FULL_GROUP_BY

0x00000040

MODE_NO_UNSIGNED_SUBTRACTION

0x00000080

MODE_NO_DIR_IN_CREATE

0x00000100

MODE_POSTGRESQL

0x00000200

MODE_ORACLE

0x00000400

MODE_MSSQL

0x00000800

MODE_DB2

0x00001000

MODE_MAXDB

0x00002000

MODE_NO_KEY_OPTIONS

0x00004000

MODE_NO_TABLE_OPTIONS

0x00008000

MODE_NO_FIELD_OPTIONS

0x00010000

MODE_MYSQL323

0x00020000

MODE_MYSQL40

0x00040000

MODE_ANSI

0x00080000

MODE_NO_AUTO_VALUE_ON_ZERO

0x00100000

MODE_NO_BACKSLASH_ESCAPES

0x00200000

MODE_STRICT_TRANS_TABLES

0x00400000

MODE_STRICT_ALL_TABLES

0x00800000

MODE_NO_ZERO_IN_DATE

0x01000000

MODE_NO_ZERO_DATE

0x02000000

MODE_INVALID_DATES

0x04000000

MODE_ERROR_FOR_DIVISION_BY_ZERO

0x08000000

MODE_TRADITIONAL

0x10000000

MODE_NO_AUTO_CREATE_USER

0x20000000

MODE_HIGH_NOT_PRECEDENCE

0x40000000

MODE_NO_ENGINE_SUBSTITUTION

0x80000000

MODE_PAD_CHAR_TO_FULL_LENGTH

Q_CATALOG_NZ_CODE (0x02):

  • uint<1> length

  • string<length +1> catalog name + '\0'

Q_AUTO_INCREMENT (0x03):

  • uint<2> auto_increment increment

  • uint<2> auto_increment offset

Q_CHARSET_CODE (0x04):

  • uint<2> client character set

  • uint<2> collation_connection

  • uint<2> collation_server

Q_TIMEZONE_CODE (0x05):

  • uint<1> length

  • string time zone

Q_CATALOG_NZ_CODE (0x06):

  • uint<1> length

  • string catalog

Q_LC_TIME_NAMES_CODE (0x07):

  • uint<2> code The mapping between code and names are defined in sql_locale.cc.

Q_CHARSET_DATABASE_CODE (0x08)

  • uint<2> database collation

Q_TABLE_MAP_FOR_UPDATE_CODE (0x09)

  • uint<8> table bittmask Every bit of this variable represents a table, and is set to 1 if the corresponding table is to be updated by this statement.

Q_MASTER_DATA_WRITTEN_CODE (0x0A): #not in use anymore

  • uint<4> original event length

Q_INVOKER (0x0B|):

  • uint<1> user name length

  • string user name

  • uint<1> host name length

  • string host name

Q_UPDATED_DB_NAMES (0x0C): MySQL only

  • uint<1> count

  • for (i=0;i < count; i++)

    • string Null terminated database name

Q_MICROSECONDS (0x0D): MySQL only

  • uint<3> microsecond part

Q_HRNOW (0x80): MariaDB only

  • uint<3> microsecond part

Q_XID (0x81): MariaDB only

  • uint<8> xid

Example With CRC32

71 17 28 5a 02 8c 27 00  00 55 00 00 00 01 09 00  q.(Z..'..U......
00 00 00 66 01 00 00 00  00 00 00 00 00 00 1a 00  ...f............
00 00 00 00 00 01 00 00  00 50 00 00 00 00 06 03  .........P......
73 74 64 04 08 00 08 00  08 00 00 54 52 55 4e 43  std........TRUNC
41 54 45 20 54 41 42 4c  45 20 74 65 73 74 2e 74  ATE TABLE test.t
34 4a 69 9e ed                                    4Ji..

Header, 19 Bytes

  • Event Time[4] = 71 17 28 5a ===> 1512576881

  • Event Type[1] = 2

  • Server_id[4] = 8c 27 00 00 ===> 10124

  • Event Size = 55 00 00 00 ===> 85

  • Next Pos = 01 09 00 00 ===> 2305

  • Flags = 00 00 => 0

Content, Variable Data

  • Thread ID[4] = 66 01 00 00 ===> 358

  • Execution Time[4] = 00 00 00 00 => 0 seconds

  • Statement default database name len[1] = 00 => 0 (no default db)

  • Error code[2] = 00 00 => 0 (no errors)

  • Status variable block len[2] = 1a 00 => 26

  • Status variables[n] = 00 ... 08 00

  • The default database[string] = 00 = 0 (no default db)

  • The SQL statement[string] = TRUNCATE TABLE test.t4

CRC32, 4 Bytes

  • 4a 69 9e ed

Example With Default db and CRC32

MariaDB []> use test;
Database changed
MariaDB [test]> TRUNCATE TABLE t4;
...

ce 22 28 5a 02 8c 27 00  00 54 00 00 00 87 0c 00  ."(Z..'..T......
00 00 00 66 01 00 00 01  00 00 00 04 00 00 1a 00  ...f............
00 00 00 00 00 01 00 00  00 50 00 00 00 00 06 03  .........P......
73 74 64 04 08 00 08 00  08 00 74 65 73 74 00 54  std.......test.T
52 55 4e 43 41 54 45 20  54 41 42 4c 45 20 74 34  RUNCATE TABLE t4
08 f1 09 16                                       ....

Content, Variable Data

  • Thread ID[4] = 66 01 00 00 ===> 358

  • Execution Time[4] = 10 00 00 00 => 1 second

  • Statement default database name len[1] = 04 => 4 (default db is "test")

  • Error code[2] = 00 00 => 0 (no errors)

  • Status variable block len[2] = 1a 00 => 26

  • Status variables[n] = 00 ... 08 00

  • The default database[string] = 74 65 73 74 00 =>test

  • The SQL statement[string] = TRUNCATE TABLE test.t4

This page is licensed: CC BY-SA / Gnu FDL

RAND_EVENT

The SQL function RAND() generates a random number.

A RAND_EVENT contains two seed values that set the rand_seed1 and rand_seed2 system variables that are used to compute the random number.

Note: it is written only before a QUERY_EVENT and is not used with row-based logging.

Header

  • Event Type is 13 (0x0d)

Fields

  • uint<8> The value for the first seed

  • uint<8> The value for the second seed

Example From mysqlbinlog Utility, No CRC32

# at 389
#171206 13:46:56 server id 10116  end_log_pos 424 	Rand
SET @@RAND_SEED1=685157301, @@RAND_SEED2=758850369/*!*/;
# at 424

Example Event As It's Written In The Binlog File

c0 e6 27 5a 0d 84 27 00 00 23 00 00 00 a8 01 00 ..'Z..'..#...... 00 00 00 b5 ab d6 28 00 00 00 00 41 23 3b 2d 00 ......(......... 00 00 00 ....

This page is licensed: CC BY-SA / Gnu FDL

ROTATE_EVENT

When a binary log file exceeds the configured size limit, a ROTATE_EVENT is written at the end of the file, pointing to the next file in the sequence.

ROTATE_EVENT is generated locally and written to the binary log on the master and it's also written when a FLUSH LOGS statement occurs on the master server.

The ROTATE_EVENT is sent to the connected slave servers.

Header

  • The Event Type is set ROTATE_EVENT (0x4)

Fields

  • uint<8> The position of the first event in the next log file. Note: it always contains the number 4 (meaning the next event starts at position 4 in the next binary log).

  • string The next binary log name. The filename is not null-terminated.

Example of Transmission With CRC32 (The Last 4 Bytes)

T 127.0.0.1:8808 -> 127.0.0.1:57157 [AP]
  30 00 00 4d 00 bc 4e 21    5a 04 d9 27 00 00 2f 00    0..M..N!Z..'../.
  00 00 c0 01 00 00 00 00    04 00 00 00 00 00 00 00    ................
  6d 79 73 71 6c 2d 62 69    6e 2e 30 30 30 30 31 39    mysql-bin.000019
  b2 bc db bf                                           ....

This page is licensed: CC BY-SA / Gnu FDL

ROWS_EVENT_V1/V2, ROWS_COMPRESSED_EVENT_V1

A ROWS_EVENT_V1 is written for row based replication if data is inserted, deleted or updated

A ROWS_EVENT (version 2) is written for row based replication if data is inserted, deleted or updated if database server is MySQL 5.6 or newer. MariaDB Server doesn't send version 2 row events.

Event types

WRITE_ROWS_EVENT_V1

Insert new row

UPDATE_ROWS_EVENT_V1

Update existing row

DELETE_ROWS_EVENT_V1

Delete existing row

WRITE_ROWS_COMPRESSED_EVENT_V1

Insert new row

UPDATE_ROWS_COMPRESSED_EVENT_V1

Update existing row

DELETE_ROWS_COMPRESSED_EVENT_V1

Delete existing row

WRITE_ROWS_EVENT

Insert new row (version 2, MySQL only)

UPDATE_ROWS_EVENT

Update existing row (version 2, MySQL only)

DELETE_ROWS_EVENT

Delete existing row (version 2, MySQL only)

Header

  • WRITE_ROWS_EVENT_V1: Event Type is 23 (0x17)

  • UPDATE_ROWS_EVENT_V1: Event Type is 24 (0x18)

  • DELETE_ROWS_EVENT_V1: Event Type is 25 (0x19)

  • WRITE_ROWS_EVENT: Event Type is 30 (0xFD)

  • UPDATE_ROWS_EVENT: Event Type is 31 (0xFE)

  • DELETE_ROWS_EVENT: Event Type is 32 (0x20)

  • WRITE_ROWS_COMPRESSED_EVENT_V1: Event Type is 166 (0xA6)

  • UPDATE_ROWS_COMPRESSED_EVENT_V1: Event Type is 167 (0xA7)

  • DELETE_ROWS_COMPRESSED_EVENT_V1: Event Type is 168 (0xA8)

Fields

  • uint<6> The table id

  • uint<2> Flags

  • if rows_event is version 2

    • uint<2> Extra data length

    • string Extra data

  • uint Number of columns

  • byteColumns used. n = (number_of_columns + 7)/8

  • if (event_type == UPDATE_ROWS_EVENT_v1

    • byte Columns used (Update). n = (number_of_columns + 7)/8

  • if *_COMPRESSED_EVENT_V1

    • byte<1> header

      • algorithm: (header & 0x07) >> 4 (always 0=zlib)

      • header_size: header & 0x07

    • byte<header_size>uncompressed length, stored in MyISAM format:

  • byte Null Bitmap (n = (number_of_columns + 7)/8)

  • string Column data. The length needs to be calculated by checking the column types from referring TABLE_MAP_EVENT.

  • if (event_type == UPDATE_ROWS_EVENT_v1

    • byte Null Bitmap_Update. n = (number_of_columns + 7)/8

    • string Update Column data. The length needs to be calculated by checking the used colums bitmap and column types from referring TABLE_MAP_EVENT.

Table id

Table id refers to a table defined by TABLE_MAP_EVENT. The special value 0xFFFFFF should have "end of statement flag" (0x0001) set and indicates that table maps can be freed.

Flags

0x0001

End of statement

0x0002

No foreign key checks

0x0004

No unique key checks

0x0008

Indicates that rows in this event are complete

0x0010

No check constraints

Extra data length (version 2)

The length of extra data

Extra data (version 2)

Extra data, length is extra data length - 2

Column Data Formats

The row data is stored in a packed format where each field is encoded in a particular format. The encoding is almost identical to the binary protocol but there are a few differences.

The field metadata is stored in the metadata block of the TABLE_MAP_EVENT. The metadata is required to decode the events. The following list shows number of bytes a field uses from the metadata block.

  • 2 bytes

    • MYSQL_TYPE_BIT

    • MYSQL_TYPE_ENUM

    • MYSQL_TYPE_SET

    • MYSQL_TYPE_NEWDECIMAL

    • MYSQL_TYPE_DECIMAL

    • MYSQL_TYPE_VARCHAR

    • MYSQL_TYPE_VAR_STRING

    • MYSQL_TYPE_STRING

  • 1 byte

    • MYSQL_TYPE_TINY_BLOB

    • MYSQL_TYPE_MEDIUM_BLOB

    • MYSQL_TYPE_LONG_BLOB

    • MYSQL_TYPE_BLOB

    • MYSQL_TYPE_TIMESTAMP2

    • MYSQL_TYPE_DATETIME2

    • MYSQL_TYPE_TIME2

    • MYSQL_TYPE_FLOAT

    • MYSQL_TYPE_DOUBLE

The types that aren't listed here do not store data in the metadata block.

Simple Types

MYSQL_TYPE_NULL

Bit set in null bitmap, no value in row data

MYSQL_TYPE_TINY

1 byte integer

MYSQL_TYPE_YEAR

1 byte integer (year = value + 1900)

MYSQL_TYPE_SHORT

2 byte integer

MYSQL_TYPE_INT24

3 byte integer

MYSQL_TYPE_LONG

4 byte integer

MYSQL_TYPE_LONGLONG

8 byte integer

MYSQL_TYPE_FLOAT

4 byte floating point value (stored as a C float type)

MYSQL_TYPE_DOUBLE

8 byte floating point value (stored as a C double type)

MYSQL_TYPE_BLOB and other blob types

Stored as a length-encoded string where the string is preceded by a variable-sized integer that stores the length of the blob. The size of the preceding integer in bytes is stored as a one byte integer in the table metadata that is a part of the table map event.

For example if the value 4 is stored in the table metadata the length is stored as a 4 byte integer (e.g. uint32_t) followed by the data.

The exact column_type can be determined by the metadata length:

Length

Type

1

MYSQL_TYPE_TINY_BLOB

2

MYSQL_TYPE_BLOB

3

MYSQL_TYPE_MEDIUM_BLOB

4

MYSQL_TYPE_LONG_BLOB

MYSQL_TYPE_STRING, MYSQL_TYPE_SET and MYSQL_TYPE_ENUM

Stored as a fixed-length string with the length of the string stored in the second byte of the table metadata. All three of these types are stored as MYSQL_TYPE_STRING in the binlog and the real type of the field is stored in the first byte of the metadata.

MYSQL_TYPE_VARCHAR and other variable length string types

Stored as a length-encoded string where the string is preceded by a variable-sized integer that stores the length of the string. The field length is stored as a two byte integer in the table metadata.

If the field length is larger than 255, the string length is stored as a two byte integer. If the value is equal to or less than 255, the string length is stored as a one byte integer.

MYSQL_TYPE_DATETIME

Note: This field type is only used in MariaDB if global variable mysql56_temporal_format was set to OFF. Stored as a 8 byte value with the values stored as multiples of 100. This means that the stored value is in the format YYYYMMDDHHMMSS and can be easily extracted by repeatedly calculating the remainder of dividing the value by 100 and dividing the value by 100. The following pseudo-code demonstrates extracting the value.

value = read_8_byte_value(row_data)
date_val= value / 1000000
time_val= value % 1000000

year = (date_val / 100) / 100
month = (date_val / 100) % 100
day = date_val % 100
hour= (time_val / 100) / 100
minute = (time_val / 100) % 100
second = time_val % 100

MYSQL_TYPE_TIME

Note: This field type is only used in MariaDB if global variable mysql56_temporal_format was set to OFF. Stored as a 3 byte value with the values stored as multiples of 100. This means that the stored value is in the format HHMMSS and can be easily extracted the same way a MYSQL_TYPE_DATETIME is extracted. The following pseudo-code demonstrates extracting the value.

time_val = read_3_byte_value(row_data); /* myisam pack format */
hour= (time_val / 100) / 100;
minute = (time_val / 100) % 100;
second = time_val % 100;

MYSQL_TYPE_DATETIME2

Stored as 4-byte value The number of decimals for the fractional part is stored in the table metadata as a one byte value. The number of bytes that follow the 5 byte datetime value can be calculated with the following formula: (decimals + 1) / 2

val = read_5_byte_value(row_data) - 0x8000000000
d_val= val >> 17;
t_val = val % (1 << 17);

day= d_val % (1 << 5);
month= (d_val >> 5) % 13;
year= (d_val >> 5) / 13;
second= t_val % (1 << 6);
minute= (t_val >> 6) % (1 << 6);
hour= (t_val)(time_part >> 12);

MYSQL_TYPE_TIME2

Stored as 3-byte value The number of decimals for the fractional part is stored in the table metadata as a one byte value. The number of bytes that follow the 3 byte time value can be calculated with the following formula: (decimals + 1) / 2

t_val = read_3_byte_value(row_data) - 0x800000
if (t_val < 0)
{
  signed= 1;
  t_val= - tval;
}
hour= (t_val >> 12) % (1 << 10);
minute= (t_val >> 6) % (1 << 6);
second= t_val % (1 << 6);

MYSQL_TYPE_TIMESTAMP2

Stored as a 4 byte UNIX timestamp (number of seconds since 00:00, Jan 1 1970 UTC) followed by the fractional second parts. The number of decimals for the fractional part is stored in the table metadata as a one byte value. The number of bytes that follow the 4 byte timestamp can be calculated with the following formula: (decimals + 1) / 2

Microseconds for MYSQL_TYPE_DATETIME2, MYSQL_TYPE_TIME2 and MYSQL_TYPE_TIMESTAMP

len = (decimals + 1) / 2
  val= read_len_byte_value(row_data)
  llen= 0: microseconds= 0
  len = 1 or 2: microseconds = val * 10000
  len = 3 or 4: microseconds = 2-bytes val in myisam pack format
  len = 5 or 6: microseconds = 3-bytes val in myisam pack format

MYSQL_TYPE_DATE

Stored as a 3 byte value where bits 1 to 5 store the day, bits 6 to 9 store the month and the remaining bits store the year.

MYSQL_TYPE_TIMESTAMP

Stored as a 4 byte UNIX timestamp (number of seconds since 00:00, Jan 1 1970 UTC).

Example From mysqlbinlog Utility, CRC32

# at 1680
#180611  9:50:51 server id 1  end_log_pos 1754 CRC32 0x5415a8fb 	Write_rows: table id 23 flags: STMT_END_F

BINLOG '
2ykeWxMBAAAAPgAAAJAGAAAAABcAAAAAAAEABHRlc3QACWJ1bGtfbnVsbAAFDwMFE/YGFAAIAAMB
H1bULg8=
2ykeWxcBAAAASgAAANoGAAAAABcAAAAAAAEABf/gATMDAAAAAAAAAAAACECAAACDAP/gATMDAAAA
AAAAAAAACECAAACDAPuoFVQ=
'/*!*/;

Example Event As It's Written In The Binlog File

db 29 1e 5b 17 01 00 00 00 4a 00       .).[.....J.
00 00 da 06 00 00 00 00 17 00 00 00 00 00 01 00  ................
05 ff e0 01 33 03 00 00 00 00 00 00 00 00 00 08  ....3...........
40 80 00 00 83 00 ff e0 01 33 03 00 00 00 00 00  @........3......
00 00 00 00 08 40 80 00 00 83 00 fb a8 15 54     .....@........T                                   ....

This page is licensed: CC BY-SA / Gnu FDL

START_ENCRYPTION_EVENT

The START_ENCRYPTION event is written to every binary log file if encrypt_binlog is set to ON.

This event is written just once, after the Format Description event (which is the first event of a binlog file at pos 4).

The event has the 19 bytes event header with EventType set to value 164 (0xa4) + 17 bytes data.

Header

  • Event type is 164 (0xa4)

Fields

  • uint<1> The Encryption scheme, always set to 1 for system files.

  • uint<4> The Encryption key version.

  • byte<12> Nonce (12 random bytes) of current binlog file.

Decryption of following events

All data of following events in the binlog file are encrypted, except for the event_length field

The 16 byte encryption IV is generated from the 12 byte nonce (uint<12>) in the binlog plus the current position of the event being encrypted (uint<4>). This means the last four bytes of the IV change for every event and the first 12 bytes change for every binlog file.

Since the event_length is always unencrypted, the encrypted data block has to be modified before it can be decrypted:

  • store event_length

  • copy the first four bytes (encrypted timestamp) to event_length position (offset=9)

  • decrypt starting from offset 4 and store result at offset 4 of decrypted buffer

The unencrypted block now also needs to be modified

  • move unencrypted timestamp value from offset 9 to the beginning (offset=0)

  • store event_length at position 9

Complete example with CRC32 from a binary log.

b8 5f 5a 59 a4 5d 00 00  00 28 00 00 00 21 01 00 ._ZY.]...(...!..
00 00 00 01 01 00 00 00  65 57 50 26 63 59 37 46 ........eWP&cY7F
2f 3b 33 23 06 bb da 62                          /;3#...b

header, 19 bytes:

  • timestamp [4] = b8 5f 5a 59 => 59 5a 5f b8 => 1499094968 [2017-07-03 17:16:08]

  • type [1} = a4 => 164

  • server_id [4} = 5d 00 00 00 => 00 00 00 5d => 93

  • event_size [4] = 28 00 00 00 => 00 00 00 28 => 40 [header + content + crc32(header + content)]

  • next_pos [4] = 21 01 00 00 => 00 00 01 21 => 289

  • flags [2] = 00 00 => 0

content,17 bytes

  • Enc scheme [1] = 01 => 1

  • Enc key ver [4] = 01 00 00 00 => 00 00 00 01 => 1

  • Nonce [12] = eWP&cY7F/;3#

crc32, 4 bytes, of the whole event (header[19] + content[17])

  • 06 bb da 62 => 62 da bb 06 => 1658501894

This page is licensed: CC BY-SA / Gnu FDL

STOP_EVENT

The master server writes the event to the binary log when it shuts down or when resuming after a mysqld process crash.

A new binary log file is always created but there is no ROTATE_EVENT.

STOP_EVENT is then the last written event after clean shutdown or resuming a crash.

This event is never sent to slave servers.

Header

  • Event header with EventType set to STOP_EVENT (0x03).

  • Event header NextPos set to EOF

  • No special flags added.

Fields

  • The event has no data

Example With CRC32 (Last 4 Bytes)

Event size = header[19] + 0 bytes data + 4 CRC32 = 23

3a b8 15 5a 03 01 00 00  00 17 00 00 00 09 0c 00  ..Z............
00 00 00 4e 99 ee 2c                              ...N..,

This page is licensed: CC BY-SA / Gnu FDL

TABLE_MAP_EVENT

Used for row-based binary logging beginning (binlog_format=ROW or MIXED)

This event precedes each row operation event and maps a table definition to a number, where the table definition consists of database and table names.

Header

  • Event Type is 19 (0x13)

Fields

Fixed data part:

  • uint<6> The table ID.

  • uint<2> Reserved for future use.

Variable data part:

  • uint<1> Database name length.

  • string The database name (null-terminated).

  • uint<1> Table name length.

  • string The table name (null-terminated).

  • int The number of columns in the table.

  • byte An array of 'n' column types, one byte per column.

  • int The length of the metadata block.

  • byte The metadata block;

  • byte Bit-field indicating whether each column can be NULL, one bit per column.

  • if (more_data_available)

    • byte Optional metadata block;

Metadata block

The metadata block contains type specific metadata information for each column

Type

Length

Description

MYSQL_TYPE_BLOB

1

Number of bytes for length: e.g. 4 bytes means length is stored in a 4 byte integer)

MYSQL_TYPE_DATETIME2

1

Length of microseconds

MYSQL_TYPE_DECIMAL

2

Not in use anymore

MYSQL_TYPE_DOUBLE ,MYSQL_TYPE_FLOAT

1

length (4 or 8 bytes

MYSQL_TYPE_STRING

2

1st byte contains type (MYSQL_TYPE_STRING, MYSQL_TYPE_ENUM, or MYSQL_TYPE_SET, 2nd byte contains length

MYSQL_TYPE_NEWDECIMAL

2

Precision, Scale

MYSQL_TYPE_TIME2

1

Length of microseconds

MYSQL_TYPE_TIMESTAMP2

1

Length of microseconds

MYSQL_TYPE_VARCHAR, MYSQL_TYPE_VAR_STRING

2

Defined varchar length. If the value is > 255, length is stored in 2 bytes, otherwise 1 byte

Optional metadata block

Optional metadata will be available if global server variable BINLOG_ROW_METADATA was set to values MIN or FULL.

The metadata block consists of one or more of the following blocks:

  • byte<1> Optional metadata type

  • int Length

  • byte Data

Optional metadata types:

Name

Value

Mode

Description

SIGNEDNESS

1

MIN

Data contains a bitmap indicating which integer columns are signed

DEFAULT_CHARSET

2

MIN

Character set of string columns, used if most columns have the same result. Columns with other character sets will follow as pair (column_index, collation number).

COLUMN_CHARSET

3

MIN

Character set of columns, used if columns have different character sets. Returned as a sequence of collation numbers.

COLUMN_NAME

4

FULL

List of Column names, the first byte specifies the length of the column name

SET_STR_VALUE

5

FULL

List of set values: First byte is the number of different values, followed by length/value pairs.

ENUM_STR_VALUE

6

FULL

Same as SET_STR_VALUE. Since ENUM values might have up to 0xFFFF members, the number of values is a length encoded integer.

GEOMETRY_TYPE

7

FULL

A sequence of bytes repesenting the type of GEOMETRY columns: 0 = GEOMETRY, 1 = POINT, 2 = LINESTRING, 3 = POLYGON, 4=MULTIPOINT, 5 = MULTILINESTRING, 6 = MULTIPOLYGON, 7 = GEOMETRYCOLLECTION

SIMPLE_PRIMARY_KEY

8

FULL

A sequence of length encoded column indexes.

PRIMARY_KEY_WITH_PREFIX

9

FULL

A sequence of length encoded column indexes and prefix lengths.

ENUM_AND_SET_DEFAULT_CHARSET

10

FULL

The default character set number used for ENUM and SET columns

ENUM_AND_SET_COLUMN_CHARSET

11

FULL

Character set of ENUM and SET columns, used if these columns have different character sets. Returned as a sequence of collation numbers.

Example From mysqlbinlog

# at 847
#171206 13:43:00 server id 10124  end_log_pos 892 CRC32 0xbe3c6b05 	Table_map: `test`.`t4` mapped to number 33
# at 892

Complete Event

d4 e5 27 5a 13 8c 27 00  00 2d 00 00 00 7c 03 00  ..'Z..'..-...|..
00 00 00 21 00 00 00 00  00 01 00 04 74 65 73 74  ...!........test
00 02 74 34 00 01 03 01  01 05 6b 3c be           ..t4......k<.

This page is licensed: CC BY-SA / Gnu FDL

USER_VAR_EVENT

A USER_VAR_EVENT is written every time a statement uses a user defined variable.

Header

  • Event Type is 14 (0x0e)

Fields

  • uint<4> The length of user variable name

  • string The name of the user variable

  • uint<1> NULL indicator

  • if (not null indicator)

    • uint<1> variable type

    • uint<4> collation number

    • uint<4> The length of value

    • string value

    • uint<1> flags

Variable type

Value

Type

Example

0x00

STRING_RESULT

set @a:="foo"

0x01

REAL_RESULT

set @a:= @@timestamp

0x02

INT_RESULT

set @a:= 4

0x03

ROW_RESULT

(not in use)

0x04

DECIMAL_RESULT

set @a:=1.2345

Flag

0x01

unsigned

Example for SET @foo:="bar"from mysqlbinlog Utility, CRC32

# at 511
#180610 10:26:43 server id 1  end_log_pos 554 CRC32 0x7dd93d6b 	User_var
SET @`foo`:=_utf8 X'626172' COLLATE `utf8_general_ci`/*!*/;

Example Event As It's Written In The Binlog File

c3 e0 1c 5b 0e 01 00 00 00 2b 00       ...[.....+.
00 00 2a 02 00 00 00 00 03 00 00 00 66 6f 6f 00  ..*.........foo.
00 21 00 00 00 03 00 00 00 62 61 72 6b 3d d9 7d  .!.......bark=.}                                          ....

This page is licensed: CC BY-SA / Gnu FDL

XA_PREPARE_LOG_EVENT

A XA_PREPARE_LOG_EVENT records the prepare phase of a distributed transaction using the XA log. It is used to ensure atomicity and consistency of transactions.

Header

  • Event Type is XA_PREPARE_LOG_EVENT (0x26)

Fields

  • uint<1> One Phase Commit

  • uint<4> Format ID

  • uint<4> Length of gtrid

  • uint<1> Length of bqual

Payload:

  • byte xid, where n is sum of gtrid and bqual lengths

This page is licensed: CC BY-SA / Gnu FDL

XID_EVENT

An XID event is generated for a COMMIT of a transaction that modifies one or more tables of an XA-capable storage engine.

Header

  • Event Type is XID_EVENT (0x10)

Fields

uint<8> The XID transaction number.

Complete Example With CRC32

ee b7 15 5a 10 01 00 00  00 1f 00 00 00 f2 0b 00  ...Z............
00 00 00 66 00 00 00 00  00 00 00 09 30 45 a8     ...f.........0E.

Header, 19 Bytes

Event size is: header[19] + XID[8] + CRC32[4] of (header + xid)

  • Event Time ee b7 15 5a => 5a 15 b7 ee => 1511372782 [2017-11-22 18:46:22]

  • Event Type = 10

  • Server_id 01 00 00 00 => 1

  • Event Size 1f 00 00 00 => 31

  • Next Pos f2 0b 00 00 => 00 00 0b f2 => 3058

  • Flags 00 00 = 0

Content, 8 Bytes

  • XID 66 00 00 00 00 00 00 00 => 102

CRC32, 4 Bytes

  • 09 30 45 a8 => a8 45 30 09 => 2823106569

This page is licensed: CC BY-SA / Gnu FDL