Features overview // Example // Use-cases // Getting started // Project status
Piqi is a set of languages and open-source tools for working with structured data. It includes:
-
A cross-language data serialization system compatible with Google Protocol Buffers. It allows programs implemented in various languages to exchange and persist data in a portable manner.
-
Piq — a human-friendly typed data representation language. It is designed to be more convenient for representing, viewing and editing data than JSON, XML, CSV, S-expressions and other formats.
-
Piqi — a powerful data definition language. It is specially designed to be used with Piq, but also works as a schema language for other data formats including JSON, XML and Protocol Buffers binary format.
-
Tools for validating, pretty-printing and converting data between Piq, JSON, XML and Protocol Buffers formats.
-
Piqi-RPC — an RPC-over-HTTP system for Erlang. It provides a simple way to expose Erlang services via JSON, XML and Google Protocol Buffers over HTTP.
As a data serialization system, Piqi implements native support for OCaml and Erlang. Connectivity with other programming languages is provided via Google Protocol Buffers. Overall, Piqi provides a more natural mapping to functional programming languages compared to various serialization systems that were originally designed for imperative or object-oriented languages.
Piqi was inspired by Google Protocol Buffers and specially designed to be largely compatible with it. Like Protocol Buffers, Piqi relies on type definitions and supports data schema evolution. The main difference is that Piqi has a richer data model, high-level modules and a powerful data representation language (Piq).
The combination of data representation (Piq) and data definition (Piqi) languages is similar to the concept of "valid XML" (i.e. XML conforming to some XML Schema). However, unlike XML, Piq has a concise, clean syntax and a data model similar to those of high-level programming languages.
More information about how Piqi compares to other systems can be found on the Rationale and FAQ pages. Compatibility between Protocol Buffers and Piqi is described here.
Features overview
Below is a high-level overview of individual Piqi components and their features.
- Piq — a data representation language (
.piqfile extension) -
Piq is a text-based language that allows humans to conveniently read, write and edit structured data.
It has expressive and concise syntax for representing various data objects such as numbers, booleans, strings, binaries (byte arrays), lists, verbatim text, etc.
Piq is a stream format, which means that one
.piqfile can contain multiple Piq objects. In addition to data itself, Piq streams can also include type definitions in a form of embedded Piqi modules. - Piqi — a data definition language (
.piqifile extension) -
Piqi is a data definition language for Piq and its encodings. It stands for "Piq Interfaces".
Piqi supports definition of records, variants (also known as tagged unions), enums, lists and type aliases on top of a rich set of built-in data types.
Piqi supports high-level modules and mechanisms for importing and including definitions from other modules.
Piqi type definitions are extensible which allows schema evolution while maintaining backward and forward compatibility.
The combination of rich data model, extensibility and module inclusion allows Piqi language to define itself in a series of type specifications. Such self-definition property is interesting by itself, but it is also utilized for implementing Piqi and makes Piqi language extensible out of the box.
Piqi modules and types (defined in
.piqifiles) can be converted to Google Protocol Buffers type specifications (.protofiles) and vice versa. - Alternative encodings for Piq data (
.piq.wireand.piq.jsonfile extensions) -
Piq streams and individual Piq objects (values) can be converted between text-based Piq representation, binary, JSON and XML formats using Piqi tools (see below).
While Piq is designed to be used by humans, binary encoding provides compact and portable representation for structured data. Binary-encoded data can be more efficiently processed by programs, stored, and transferred over the network.
Piqi uses the same binary encoding as in Google Protocol Buffers.
- Support for programming languages
-
Piqi provides static mappings to programming languages either through Google Protocol Buffers or by using its own Piq interface compiler (
piqic) and runtime libraries for OCaml and Erlang.Protocol Buffers officially support Python, Java and C++. There are third-party mappings for many other languages, such as C#, Go, Common Lisp, Scala, etc.
Programs written in dynamic languages such as JavaScript can work with data encoded in JSON or XML formats.
- Command-line tools for working with Piq and Piqi
-
piqiis a single executable program with command-line interface. It implements a set of subcommands. For example:piqi convert— converts structured data between Piq, JSON, XML and binary formats.piqi check— checks.piqand.piqivaliditypiqi pp— pretty-prints.piqand.piqifilespiqi to-proto,piqi to-proto— converts.piqito/from.proto
- Piqi-RPC: an RPC-over HTTP system for Erlang
-
Piqi-RPC gives Erlang developers a convenient and reliable way to integrate services written in Erlang with non-Erlang clients using JSON, XML or Google Protocol Buffers over HTTP.
It also includes
piqi call— a native client that allows to call Piqi-RPC Erlang functions remotely using command-line interface.piqi callinterprets command-line arguments as input parameters for functions.Piqi-RPC is based on Piqi data serialization for Erlang and Piqi data conversion functionality.
Example
Below are two clickable examples:
-
person.piqis a Piq file that represents an addressbook containing two person records. -
person.piqiis a Piqi module that contains definition of the person record type.
Detailed description of the tabs’ contents and other examples can be found on Examples page.
% definition of a record type
.record [
.name person % record name
.field [
.name name % field name
.type string % field type
]
.field [
.name id
.type int
]
.field [
.name email
.type string
.optional % field is optional
]
.field [
.name phone
.type phone-number
.repeated % field can be repeated 0 or more times
]
]
.record [
.name phone-number
.field [
.name number
.type string
]
.field [
.name type
.type phone-type
.optional
.default.home % default value for an optional field
% ".default.home" is a shorthand for ".default (.home)", where
% ".home" is a value of the enum type defined below
]
]
% definition of an enum type
.enum [
.name phone-type
.option [ .name mobile ]
.option [ .name home ]
.option [ .name work ]
]
message person {
required string name = 1;
required sint32 id = 2;
optional string email = 3;
repeated phone_number phone = 4;
}
message phone_number {
required string number = 1;
optional phone_type type = 2 [default = home];
}
enum phone_type {
mobile = 1;
home = 2;
work = 3;
}
module person
type person =
{
- name :: string()
- id :: int()
? email :: string()
* phone :: phone-number()
}
type phone-number =
{
- number :: string()
? type :: phone-type() = .home
}
type phone-type =
| mobile
| home
| work
% this is an object of type "person" which is defined in module "person"
:person/person [
.name "J. Random Hacker"
.id 0
.email "j.r.hacker@example.com"
.phone [
.number "(111) 123 45 67"
% NOTE: phone is "home" by default
]
.phone [
.number "(222) 123 45 67"
.mobile
]
.phone [
.number "(333) 123 45 67"
.work
]
]
% another object of the same type
:person/person [
.name "Joe User"
.id 1
% Joe User doesn't have an email
.phone [ "(444) 123 45 67" ]
.phone [ "(555) 123 45 67" .work ]
]
:person/person [
.name "J. Random Hacker"
.id 0
.email "j.r.hacker@example.com"
.phone [ .number "(111) 123 45 67" ]
.phone [
.number "(222) 123 45 67"
.type.mobile
]
.phone [
.number "(333) 123 45 67"
.type.work
]
]
:person/person [
.name "Joe User"
.id 1
.phone [ .number "(444) 123 45 67" ]
.phone [
.number "(555) 123 45 67"
.type.work
]
]
{
"name": "J. Random Hacker",
"id": 0,
"email": "j.r.hacker@example.com",
"phone": [
{ "number": "(111) 123 45 67", "type": "home" },
{ "number": "(222) 123 45 67", "type": "mobile" },
{ "number": "(333) 123 45 67", "type": "work" }
]
}
{
"name": "Joe User",
"id": 1,
"phone": [
{ "number": "(444) 123 45 67", "type": "home" },
{ "number": "(555) 123 45 67", "type": "work" }
]
}
<?xml version="1.0" encoding="UTF-8"?>
<person>
<name>J. Random Hacker</name>
<id>0</id>
<email>j.r.hacker@example.com</email>
<phone>
<number>(111) 123 45 67</number>
<type>home</type>
</phone>
<phone>
<number>(222) 123 45 67</number>
<type>mobile</type>
</phone>
<phone>
<number>(333) 123 45 67</number>
<type>work</type>
</phone>
</person>
<?xml version="1.0" encoding="UTF-8"?>
<person>
<name>Joe User</name>
<id>1</id>
<phone>
<number>(444) 123 45 67</number>
<type>home</type>
</phone>
<phone>
<number>(555) 123 45 67</number>
<type>work</type>
</phone>
</person>
:piqi [
.module person
.record [
.name person
.field [
.name name
.type string
]
.field [
.name id
.type int
]
.field [
.name email
.type string
.optional
]
.field [
.name phone
.type phone-number
.repeated
]
]
.record [
.name phone-number
.field [
.name number
.type string
]
.field [
.name type
.type phone-type
.optional
.default.home
]
]
.enum [
.name phone-type
.option [ .name mobile ]
.option [ .name home ]
.option [ .name work ]
]
]
:person/person [
.name "J. Random Hacker"
.id 0
.email "j.r.hacker@example.com"
.phone [ .number "(111) 123 45 67" ]
.phone [
.number "(222) 123 45 67"
.type.mobile
]
.phone [
.number "(333) 123 45 67"
.type.work
]
]
:person/person [
.name "Joe User"
.id 1
.phone [ .number "(444) 123 45 67" ]
.phone [
.number "(555) 123 45 67"
.type.work
]
]
0000000: 1a0d 7065 7273 6f6e 2f70 6572 736f 6e22 ..person/person"
0000010: 690a 104a 2e20 5261 6e64 6f6d 2048 6163 i..J. Random Hac
0000020: 6b65 7210 001a 166a 2e72 2e68 6163 6b65 ker....j.r.hacke
0000030: 7240 6578 616d 706c 652e 636f 6d22 110a r@example.com"..
0000040: 0f28 3131 3129 2031 3233 2034 3520 3637 .(111) 123 45 67
0000050: 2213 0a0f 2832 3232 2920 3132 3320 3435 "...(222) 123 45
0000060: 2036 3710 0122 130a 0f28 3333 3329 2031 67.."...(333) 1
0000070: 3233 2034 3520 3637 1003 2234 0a08 4a6f 23 45 67.."4..Jo
0000080: 6520 5573 6572 1002 2211 0a0f 2834 3434 e User.."...(444
0000090: 2920 3132 3320 3435 2036 3722 130a 0f28 ) 123 45 67"...(
00000a0: 3535 3529 2031 3233 2034 3520 3637 1003 555) 123 45 67..
Use-cases and limitations
Piq/Piqi can be used as a better alternative to XML or other structured data formats.
-
Piq language is designed be to more convenient for humans to write, read and edit than XML or even JSON.
-
It is also more robust than comma- or tab-separated values (CSV, TSV) formats for representing records (each record is enclosed in
[ ], and field names for required record fields can be omitted). -
Using Piqi tools, data represented in Piq format can be validated and converted to a binary form which then can be conveniently loaded into programs without any manual parsing. This operation also works in the opposite direction: programs’ binary output can be converted to a human-friendly Piq format.
In many ways this method is more robust and efficient than dynamic parsing of XML, plain JSON, CSV, S-expressions or custom formats like those used for config files.
Google Protocol Buffers users can utilize more powerful Piq language instead of built-in text format for inspecting or editing data. It is also possible to convert encoded Protobuf messages to JSON or XML formats using Piqi tools.
OCaml and Erlang programmers can use Piqi to conveniently serialize and deserialize data using a portable binary format which can be understood by programs written in Python, Java, C++ and other languages supported by Google Protocol Buffers. In this sense, Piqi provides Protocol Buffers support for OCaml and Erlang.
Piqi has certain limitations.
-
Compared to XML, Piqi doesn’t support semi-structured data and doesn’t have query and transformation languages (there are, however, plans for implementing some of these). In the meantime, it is possible to use tools such as XPath, XSLT and XQuery with Piq data represented in XML format.
-
Since Piq is a full-featured and fairly complicated language, it can’t be used from programs directly. Before processing or transferring data represented in Piq format, it needs to be converted into one of simpler and portable encodings — binary, JSON or XML.
-
Currently, Piq language implementation and Piqi tools are packaged as a standalone command-line application and not available as a library.
How to get started
Learn more about Piqi by reading documentation and going through examples.
Download or build Piqi tools and runtime libraries from sources.
Watch the Project on GitHub or subscribe to the news feed to keep up to date.
If you have questions or suggestions there is the Piqi Google Group or contact me directly at alavrik@piqi.org or at @alavrik.
If you found a bug or have any annoyances, please submit a GitHub issue.
Your feedback and participation is highly appreciated! If you find it useful in some way or another, please drop me a line.
Project status
Piqi is a BETA software.
It has been tested on Debian Linux (i386 and amd64 Lenny and Squeeze) and on Mac OS X (10.6.4). Other Unix systems on major hardware architectures should generally work. For instance, Piqi is known to work on Solaris/x86. On Windows, Piqi successfully builds under both MinGW/Msys and Cygwin. Refer to installation instructions for more details.
Check out the Roadmap page for the list of planned Piqi features.
