Features overview // Example // Use-cases // Getting started // Project status
Piqi is a set of languages and open-source tools for working with structured data. It includes:
-
A cross-language data serialization system compatible with Google Protocol Buffers. It allows programs implemented in various languages to exchange and persist data in a portable manner.
-
Piq — a human-friendly typed data representation language. It is designed to be more convenient for representing, viewing and editing data than JSON, XML, CSV, S-expressions and other formats.
-
Piqi — a universal data definition language. It can be used as a schema language for JSON, XML, Protocol Buffers and Piq.
-
Tools for validating, pretty-printing and converting data between Piq, JSON, XML and Protocol Buffers formats.
-
Piqi-RPC — an RPC-over-HTTP system for Erlang. It provides a simple way to expose Erlang services via JSON, XML and Google Protocol Buffers over HTTP.
As a data serialization system, Piqi implements native support for OCaml and Erlang. Connectivity with other programming languages is provided via Google Protocol Buffers. Overall, Piqi provides a more natural mapping to functional programming languages compared to various serialization systems that were originally designed for imperative or object-oriented languages.
Piqi was inspired by Google Protocol Buffers and specially designed to be largely compatible with it. Like Protocol Buffers, Piqi relies on type definitions and supports data schema evolution. The main difference is that Piqi has a richer data model, high-level modules and a powerful data representation language (Piq).
Combination of the data representation (Piq) and the data definition (Piqi) languages is similar to the concept of "valid XML" (i.e. XML conforming to some XML Schema). However, unlike XML, Piq has a concise, clean syntax and a data model similar to those of high-level programming languages.
More information about how Piqi compares to other systems can be found on the Rationale and FAQ pages. Compatibility between Protocol Buffers and Piqi is described here.
Features overview
Below is a high-level overview of individual Piqi components and their features.
- Piq — a data representation language (
.piqfile extension) -
Piq is a text-based language that allows humans to conveniently read, write and edit structured data.
It has expressive and concise syntax for representing various data objects such as numbers, booleans, strings, binaries (byte arrays), lists, verbatim text, etc.
Piq is a stream format, which means that one
.piqfile can contain multiple Piq objects. In addition to data, Piq streams can also include type definitions in a form of embedded Piqi modules. - Piqi — a data definition language (
.piqifile extension) -
Piqi is a data definition language for Piq and its encodings. It stands for "Piq Interfaces".
Piqi supports definition of records, variants (also known as tagged unions), enums, lists and type aliases on top of a rich set of built-in data types.
Piqi supports high-level modules and mechanisms for importing and including definitions from other modules.
Piqi type definitions are extensible which allows schema evolution while maintaining backward and forward compatibility.
The combination of rich data model, extensibility and module inclusion allows the Piqi language to define itself. Such self-definition property is interesting by itself, but it is also utilized for implementing Piqi and makes the language extensible out of the box.
Piqi modules and types (defined in
.piqifiles) can be converted to Google Protocol Buffers type specifications (.protofiles) and vice versa.Piqi modules have stable external representation in Protocol Buffers, JSON and XML formats. It allows other programs to easily and reliably work with Piqi type definitions.
- Support for programming languages
-
Piqi provides static mappings to programming languages via Google Protocol Buffers and by using its own Piqi compiler (
piqic) and runtime libraries for OCaml and Erlang.Protocol Buffers officially support Python, Java and C++. There are third-party mappings for many other languages, such as C#, Go, Common Lisp, Scala, Clojure, etc.
Programs written in dynamic languages such as JavaScript can work with data encoded in JSON or XML formats.
- Command-line tools for working with Piq and Piqi
-
piqiis a statically-linked executable program implementing a set of subcommands. For example:piqi convert— converts structured data between Piq, JSON, XML and Protocol Buffers formats.piqi check— checks.piqand.piqivaliditypiqi pp— pretty-prints.piqand.piqifilespiqi to-proto,piqi to-proto— converts.piqito/from.proto
- Piqi-RPC: an RPC-over HTTP system for Erlang
-
Piqi-RPC gives Erlang developers a convenient and reliable way to integrate services written in Erlang with non-Erlang clients using JSON, XML or Google Protocol Buffers over HTTP.
It also includes
piqi call— a native client that allows to call Piqi-RPC Erlang functions remotely using command-line interface.piqi callinterprets command-line arguments and converts them into input parameters for remote functions.
Example
Below are two clickable examples:
-
person.piqis a Piq file that represents an addressbook containing two person records. -
person.piqiis a Piqi module that contains definition of the person record type.
Detailed description of the tabs’ contents and other examples can be found on Examples page.
% definition of a record type
.record [
.name person % record name
.field [
.name name % field name
.type string % field type
]
.field [
.name id
.type int
]
.field [
.name email
.type string
.optional % field is optional
]
.field [
.name phone
.type phone-number
.repeated % field can be repeated 0 or more times
]
]
.record [
.name phone-number
.field [
.name number
.type string
]
.field [
.name type
.type phone-type
.optional
.default.home % default value for an optional field
% ".default.home" is a shorthand for ".default (.home)", where
% ".home" is a value of the enum type defined below
]
]
% definition of an enum type
.enum [
.name phone-type
.option [ .name mobile ]
.option [ .name home ]
.option [ .name work ]
]
message person {
required string name = 1;
required sint32 id = 2;
optional string email = 3;
repeated phone_number phone = 4;
}
message phone_number {
required string number = 1;
optional phone_type type = 2 [default = home];
}
enum phone_type {
mobile = 1;
home = 2;
work = 3;
}
{
"piqi_type": "piqi",
"module": "person",
"typedef": [
{
"record": {
"name": "person",
"field": [
{ "name": "name", "type": "string", "mode": "required" },
{ "name": "id", "type": "int", "mode": "required" },
{ "name": "email", "type": "string", "mode": "optional" },
{ "name": "phone", "type": "phone-number", "mode": "repeated" }
]
}
},
{
"record": {
"name": "phone-number",
"field": [
{ "name": "number", "type": "string", "mode": "required" },
{
"name": "type",
"type": "phone-type",
"mode": "optional",
"default": "home"
}
]
}
},
{
"enum": {
"name": "phone-type",
"option": [
{ "name": "mobile" },
{ "name": "home" },
{ "name": "work" }
]
}
}
]
}
<?xml version="1.0" encoding="UTF-8"?>
<value>
<module>person</module>
<typedef>
<record>
<name>person</name>
<field>
<name>name</name>
<type>string</type>
<mode>required</mode>
</field>
<field>
<name>id</name>
<type>int</type>
<mode>required</mode>
</field>
<field>
<name>email</name>
<type>string</type>
<mode>optional</mode>
</field>
<field>
<name>phone</name>
<type>phone-number</type>
<mode>repeated</mode>
</field>
</record>
</typedef>
<typedef>
<record>
<name>phone-number</name>
<field>
<name>number</name>
<type>string</type>
<mode>required</mode>
</field>
<field>
<name>type</name>
<type>phone-type</type>
<mode>optional</mode>
<default>home</default>
</field>
</record>
</typedef>
<typedef>
<enum>
<name>phone-type</name>
<option>
<name>mobile</name>
</option>
<option>
<name>home</name>
</option>
<option>
<name>work</name>
</option>
</enum>
</typedef>
</value>
type person =
{
- name :: string()
- id :: int()
? email :: string()
* phone :: phone-number()
}
type phone-number =
{
- number :: string()
? type :: phone-type() = .home
}
type phone-type =
| mobile
| home
| work
% this is an object of type "person" which is defined in module "person"
:person/person [
.name "J. Random Hacker"
.id 0
.email "j.r.hacker@example.com"
.phone [
.number "(111) 123 45 67"
% NOTE: phone is "home" by default
]
.phone [
.number "(222) 123 45 67"
.mobile
]
.phone [
.number "(333) 123 45 67"
.work
]
]
% another object of the same type
:person/person [
.name "Joe User"
.id 1
% Joe User doesn't have an email
.phone [ "(444) 123 45 67" ]
.phone [ "(555) 123 45 67" .work ]
]
:person/person [
.name "J. Random Hacker"
.id 0
.email "j.r.hacker@example.com"
.phone [ .number "(111) 123 45 67" ]
.phone [
.number "(222) 123 45 67"
.type.mobile
]
.phone [
.number "(333) 123 45 67"
.type.work
]
]
:person/person [
.name "Joe User"
.id 1
.phone [ .number "(444) 123 45 67" ]
.phone [
.number "(555) 123 45 67"
.type.work
]
]
{
"piqi_type": "person/person",
"name": "J. Random Hacker",
"id": 0,
"email": "j.r.hacker@example.com",
"phone": [
{ "number": "(111) 123 45 67", "type": "home" },
{ "number": "(222) 123 45 67", "type": "mobile" },
{ "number": "(333) 123 45 67", "type": "work" }
]
}
{
"piqi_type": "person/person",
"name": "Joe User",
"id": 1,
"phone": [
{ "number": "(444) 123 45 67", "type": "home" },
{ "number": "(555) 123 45 67", "type": "work" }
]
}
<?xml version="1.0" encoding="UTF-8"?>
<value>
<name>J. Random Hacker</name>
<id>0</id>
<email>j.r.hacker@example.com</email>
<phone>
<number>(111) 123 45 67</number>
<type>home</type>
</phone>
<phone>
<number>(222) 123 45 67</number>
<type>mobile</type>
</phone>
<phone>
<number>(333) 123 45 67</number>
<type>work</type>
</phone>
</value>
<?xml version="1.0" encoding="UTF-8"?>
<value>
<name>Joe User</name>
<id>1</id>
<phone>
<number>(444) 123 45 67</number>
<type>home</type>
</phone>
<phone>
<number>(555) 123 45 67</number>
<type>work</type>
</phone>
</value>
:piqi [
.module person
.record [
.name person
.field [
.name name
.type string
]
.field [
.name id
.type int
]
.field [
.name email
.type string
.optional
]
.field [
.name phone
.type phone-number
.repeated
]
]
.record [
.name phone-number
.field [
.name number
.type string
]
.field [
.name type
.type phone-type
.optional
.default.home
]
]
.enum [
.name phone-type
.option [ .name mobile ]
.option [ .name home ]
.option [ .name work ]
]
]
:person/person [
.name "J. Random Hacker"
.id 0
.email "j.r.hacker@example.com"
.phone [ .number "(111) 123 45 67" ]
.phone [
.number "(222) 123 45 67"
.type.mobile
]
.phone [
.number "(333) 123 45 67"
.type.work
]
]
:person/person [
.name "Joe User"
.id 1
.phone [ .number "(444) 123 45 67" ]
.phone [
.number "(555) 123 45 67"
.type.work
]
]
0000000: faff ffff 0f1c 0a09 7069 7169 2d74 7970 ........piqi-typ
0000010: 6512 0d70 6572 736f 6e2f 7065 7273 6f6e e..person/person
0000020: 1802 1269 0a10 4a2e 2052 616e 646f 6d20 ...i..J. Random
0000030: 4861 636b 6572 1000 1a16 6a2e 722e 6861 Hacker....j.r.ha
0000040: 636b 6572 4065 7861 6d70 6c65 2e63 6f6d cker@example.com
0000050: 2211 0a0f 2831 3131 2920 3132 3320 3435 "...(111) 123 45
0000060: 2036 3722 130a 0f28 3232 3229 2031 3233 67"...(222) 123
0000070: 2034 3520 3637 1001 2213 0a0f 2833 3333 45 67.."...(333
0000080: 2920 3132 3320 3435 2036 3710 0312 340a ) 123 45 67...4.
0000090: 084a 6f65 2055 7365 7210 0222 110a 0f28 .Joe User.."...(
00000a0: 3434 3429 2031 3233 2034 3520 3637 2213 444) 123 45 67".
00000b0: 0a0f 2835 3535 2920 3132 3320 3435 2036 ..(555) 123 45 6
00000c0: 3710 03 7..
Use-cases and limitations
Piq/Piqi can be used as a better alternative to XML or other structured data formats.
-
Piq language is designed be to more convenient for people to write, read and edit than XML or even JSON.
-
It is also more robust than comma- or tab-separated values (CSV, TSV) formats for representing records (each record is enclosed in
[ ], and field names for required record fields can be omitted). -
By using Piqi tools, data represented in Piq format can be validated and converted to a binary form which then can be conveniently loaded into programs without any manual parsing. This operation also works in the opposite direction: programs’ binary output can be converted to a human-friendly Piq format.
In many ways this method is more robust and efficient than dynamic parsing of XML, plain JSON, CSV, S-expressions or custom formats like those used in various config files.
Google Protocol Buffers users can utilize more powerful Piq language instead of built-in text format for inspecting or editing data. It is also possible to convert encoded Protobuf messages to JSON or XML formats using Piqi tools.
OCaml and Erlang programmers can use Piqi to conveniently serialize and deserialize data using a portable binary format that can be understood by programs written in Python, Java, C++ and other languages that implement Google Protocol Buffers support. In this sense, Piqi provides Protocol Buffers support for OCaml and Erlang.
Piqi has certain limitations.
-
Compared to XML, Piqi doesn’t support semi-structured data and doesn’t have query and transformation languages (there are, however, plans for implementing some of these). In the meantime, it is possible to use tools such as XPath, XSLT and XQuery with data represented in XML format or tools like jq for transforming and processing JSON.
-
Since Piq is a new and fairly complicated language, it can’t be used from programs directly (unlike, for example, YAML). Before processing or transferring data represented in Piq format, it needs to be converted into one of simpler and portable encodings — Protocol Buffers, JSON or XML.
-
Currently, the Piq language implementation and Piqi tools are packaged as a standalone command-line application and not available as a library.
How to get started
Learn more about Piqi by reading documentation and going through examples.
Download or build Piqi tools and runtime libraries from sources.
Watch the Project on GitHub or subscribe to the news feed.
If you have questions or suggestions there is the Piqi Google Group or contact me directly at alavrik@piqi.org or at @alavrik.
If you found a bug, please submit a GitHub issue.
Your feedback and participation is highly appreciated! If you find Piqi useful in some way or another, please drop me a line.
Project status
Piqi is a BETA software.
It has been tested on Debian Linux (i386 and amd64 Lenny and Squeeze) and on Mac OS X (10.6.4). Other Unix systems on major hardware architectures should generally work. For instance, Piqi is known to work on Solaris/x86. On Windows, Piqi successfully builds under both MinGW/Msys and Cygwin. Refer to installation instructions for more details.
Check out the Roadmap page for the list of planned Piqi features.
