Piqi (/pɪki/) is a universal schema language and a collection of tools built around it.
The Piqi language can be used to define schemas for JSON, XML, Google Protocol Buffers and some other data formats.
Some of Piqi sub-projects include:
- Command-line tools for validating, pretty-printing and converting data between JSON, XML and Protocol Buffers formats.
- A multi-format (JSON, XML, Protocol Buffers) data serialization system for Erlang and OCaml.
- Piq — a human-friendly typed data representation language. It is designed to be more convenient for viewing and editing data compared to JSON, XML, CSV, S-expressions and other formats.
- Piqi-RPC — an RPC-over-HTTP system for Erlang. It provides a simple way to expose Erlang services via JSON, XML and Protocol Buffers over HTTP.
The Piqi project was inspired by Google Protocol Buffers and designed to be largely compatible with it. Like Protocol Buffers, Piqi relies on type definitions and supports schema evolution. The main differences is that Piqi has a richer data model, high-level modules, standard mappings to JSON and XML, and comes with a powerful data representation format (Piq). Also, Piqi is a lot more extensible.
More information about how Piqi compares to other systems can be found on the Rationale and FAQ pages. Compatibility between Protocol Buffers and Piqi is described here.
Scroll down to:
Components and sub-projects
Below is a high-level overview of Piqi components and sub-projects.
- Piqi — a universal schema language (
.piqifile extension) - Piqi can be used as a schema language for JSON, XML, Protocol Bufferss and Piq.
Piqi supports definition of records, variants (also known as tagged unions), enums, lists and type aliases on top of a rich set of built-in data types.
Piqi supports high-level modules and mechanisms for importing and including definitions from other modules.
Piqi type definitions are extensible which allows schema evolution while maintaining backward and forward compatibility.
The combination of rich data model, extensibility and module inclusion allows the Piqi language to define itself. Such self-definition property is interesting by itself, but it is also utilized for implementing Piqi and makes the language extensible out of the box.
Piqi modules and types (defined in
.piqifiles) can be converted to Google Protocol Buffers type specifications (.protofiles) and vice versa.Piqi modules have stable external representation in Protocol Buffers, JSON and XML formats. It allows other programs to easily and reliably work with Piqi type definitions.
- Piq — a data representation language (
.piqfile extension) - Piq is a text-based language that allows humans to conveniently read, write and edit structured data.
It has expressive and concise syntax for representing various data objects such as numbers, booleans, strings, binaries (byte arrays), lists, verbatim text, etc.
Piq is a stream format, which means that one
.piqfile can contain multiple Piq objects. In addition to data, Piq streams can also include type definitions in a form of embedded Piqi modules. - Serialization system for programming languages
- Piqi provides static mappings to programming languages via Google Protocol Buffers and by using its own implementations for OCaml and Erlang.
Protocol Buffers officially support Python, Java and C++. There are third-party mappings for many other languages, such as C#, Go, Common Lisp, Scala, Clojure, etc.
Programs written in dynamic languages such as JavaScript can work with data encoded in JSON or XML formats.
- Command-line tools for working with Piq and Piqi
piqiis a statically-linked executable program implementing a set of subcommands. For example:piqi convert— converts structured data between Piq, JSON, XML and Protocol Buffers formats.piqi check— checks.piqand.piqivaliditypiqi pp— pretty-prints.piqand.piqifilespiqi to-proto,piqi of-proto— converts.piqito/from.proto- Piqi-RPC: an RPC-over HTTP system for Erlang
- Piqi-RPC gives Erlang developers a convenient and reliable way to integrate services written in Erlang with non-Erlang clients using JSON, XML or Google Protocol Buffers over HTTP.
It also includes
piqi call— a native client that allows to call Piqi-RPC Erlang functions remotely using command-line interface.piqi callinterprets command-line arguments and converts them into input parameters for remote functions.
Examples
Below are two clickable examples:
person.piqis a Piq file that represents an addressbook containing two person records.person.piqiis a Piqi module that contains definition of the person record type.
Detailed description of the tabs’ contents and other examples can be found on Examples page.
% definition of a record type
.record [
.name person % record name
.field [
.name name % field name
.type string % field type
]
.field [
.name id
.type int
]
.field [
.name email
.type string
.optional % field is optional
]
.field [
.name phone
.type phone-number
.repeated % field can be repeated 0 or more times
]
]
.record [
.name phone-number
.field [
.name number
.type string
]
.field [
.name type
.type phone-type
.optional
.default.home % default value for an optional field
% ".default.home" is a shorthand for ".default (.home)", where
% ".home" is a value of the enum type defined below
]
]
% definition of an enum type
.enum [
.name phone-type
.option [ .name mobile ]
.option [ .name home ]
.option [ .name work ]
]
message person {
required string name = 1;
required sint32 id = 2;
optional string email = 3;
repeated phone_number phone = 4;
}
message phone_number {
required string number = 1;
optional phone_type type = 2 [default = home];
}
enum phone_type {
mobile = 1;
home = 2;
work = 3;
}
{
"piqi_type": "piqi",
"module": "person",
"typedef": [
{
"record": {
"name": "person",
"field": [
{ "name": "name", "type": "string", "mode": "required" },
{ "name": "id", "type": "int", "mode": "required" },
{ "name": "email", "type": "string", "mode": "optional" },
{ "name": "phone", "type": "phone-number", "mode": "repeated" }
]
}
},
{
"record": {
"name": "phone-number",
"field": [
{ "name": "number", "type": "string", "mode": "required" },
{
"name": "type",
"type": "phone-type",
"mode": "optional",
"default": "home"
}
]
}
},
{
"enum": {
"name": "phone-type",
"option": [
{ "name": "mobile" },
{ "name": "home" },
{ "name": "work" }
]
}
}
]
}
<?xml version="1.0" encoding="UTF-8"?>
<value>
<module>person</module>
<typedef>
<record>
<name>person</name>
<field>
<name>name</name>
<type>string</type>
<mode>required</mode>
</field>
<field>
<name>id</name>
<type>int</type>
<mode>required</mode>
</field>
<field>
<name>email</name>
<type>string</type>
<mode>optional</mode>
</field>
<field>
<name>phone</name>
<type>phone-number</type>
<mode>repeated</mode>
</field>
</record>
</typedef>
<typedef>
<record>
<name>phone-number</name>
<field>
<name>number</name>
<type>string</type>
<mode>required</mode>
</field>
<field>
<name>type</name>
<type>phone-type</type>
<mode>optional</mode>
<default>home</default>
</field>
</record>
</typedef>
<typedef>
<enum>
<name>phone-type</name>
<option>
<name>mobile</name>
</option>
<option>
<name>home</name>
</option>
<option>
<name>work</name>
</option>
</enum>
</typedef>
</value>
type person =
{
- name :: string()
- id :: int()
? email :: string()
* phone :: phone-number()
}
type phone-number =
{
- number :: string()
? type :: phone-type() = .home
}
type phone-type =
| mobile
| home
| work
% this is an object of type "person" which is defined in module "person"
:person/person [
.name "J. Random Hacker"
.id 0
.email "j.r.hacker@example.com"
.phone [
.number "(111) 123 45 67"
% NOTE: phone is "home" by default
]
.phone [
.number "(222) 123 45 67"
.mobile
]
.phone [
.number "(333) 123 45 67"
.work
]
]
% another object of the same type
:person/person [
.name "Joe User"
.id 1
% Joe User doesn't have an email
.phone [ "(444) 123 45 67" ]
.phone [ "(555) 123 45 67" .work ]
]
:person/person [
.name "J. Random Hacker"
.id 0
.email "j.r.hacker@example.com"
.phone [ .number "(111) 123 45 67" ]
.phone [
.number "(222) 123 45 67"
.type.mobile
]
.phone [
.number "(333) 123 45 67"
.type.work
]
]
:person/person [
.name "Joe User"
.id 1
.phone [ .number "(444) 123 45 67" ]
.phone [
.number "(555) 123 45 67"
.type.work
]
]
{
"piqi_type": "person/person",
"name": "J. Random Hacker",
"id": 0,
"email": "j.r.hacker@example.com",
"phone": [
{ "number": "(111) 123 45 67", "type": "home" },
{ "number": "(222) 123 45 67", "type": "mobile" },
{ "number": "(333) 123 45 67", "type": "work" }
]
}
{
"piqi_type": "person/person",
"name": "Joe User",
"id": 1,
"phone": [
{ "number": "(444) 123 45 67", "type": "home" },
{ "number": "(555) 123 45 67", "type": "work" }
]
}
<?xml version="1.0" encoding="UTF-8"?>
<value>
<name>J. Random Hacker</name>
<id>0</id>
<email>j.r.hacker@example.com</email>
<phone>
<number>(111) 123 45 67</number>
<type>home</type>
</phone>
<phone>
<number>(222) 123 45 67</number>
<type>mobile</type>
</phone>
<phone>
<number>(333) 123 45 67</number>
<type>work</type>
</phone>
</value>
<?xml version="1.0" encoding="UTF-8"?>
<value>
<name>Joe User</name>
<id>1</id>
<phone>
<number>(444) 123 45 67</number>
<type>home</type>
</phone>
<phone>
<number>(555) 123 45 67</number>
<type>work</type>
</phone>
</value>
:piqi [
.module person
.record [
.name person
.field [
.name name
.type string
]
.field [
.name id
.type int
]
.field [
.name email
.type string
.optional
]
.field [
.name phone
.type phone-number
.repeated
]
]
.record [
.name phone-number
.field [
.name number
.type string
]
.field [
.name type
.type phone-type
.optional
.default.home
]
]
.enum [
.name phone-type
.option [ .name mobile ]
.option [ .name home ]
.option [ .name work ]
]
]
:person/person [
.name "J. Random Hacker"
.id 0
.email "j.r.hacker@example.com"
.phone [ .number "(111) 123 45 67" ]
.phone [
.number "(222) 123 45 67"
.type.mobile
]
.phone [
.number "(333) 123 45 67"
.type.work
]
]
:person/person [
.name "Joe User"
.id 1
.phone [ .number "(444) 123 45 67" ]
.phone [
.number "(555) 123 45 67"
.type.work
]
]
0000000: faff ffff 0f1c 0a09 7069 7169 2d74 7970 ........piqi-typ
0000010: 6512 0d70 6572 736f 6e2f 7065 7273 6f6e e..person/person
0000020: 1802 1269 0a10 4a2e 2052 616e 646f 6d20 ...i..J. Random
0000030: 4861 636b 6572 1000 1a16 6a2e 722e 6861 Hacker....j.r.ha
0000040: 636b 6572 4065 7861 6d70 6c65 2e63 6f6d cker@example.com
0000050: 2211 0a0f 2831 3131 2920 3132 3320 3435 "...(111) 123 45
0000060: 2036 3722 130a 0f28 3232 3229 2031 3233 67"...(222) 123
0000070: 2034 3520 3637 1001 2213 0a0f 2833 3333 45 67.."...(333
0000080: 2920 3132 3320 3435 2036 3710 0312 340a ) 123 45 67...4.
0000090: 084a 6f65 2055 7365 7210 0222 110a 0f28 .Joe User.."...(
00000a0: 3434 3429 2031 3233 2034 3520 3637 2213 444) 123 45 67".
00000b0: 0a0f 2835 3535 2920 3132 3320 3435 2036 ..(555) 123 45 6
00000c0: 3710 03 7..
Limitations
- Unlike XML, Piqi doesn’t support semi-structured data and doesn’t have query and transformation languages (there are, however, plans for implementing some of these). In the meantime, it is possible to use tools such as XPath, XSLT and XQuery with data represented in XML format or tools like jq for transforming and processing JSON.
- Currently, the Piqi language implementation and Piqi tools are packaged as a standalone command-line application and not available as a library for langauges other than Erlang and OCaml.
How to get started
Learn more about Piqi by reading documentation and going through examples.
Download or build Piqi tools from sources.
Watch the Project on GitHub or subscribe to the news feed.
If you have questions or suggestions there is the Piqi Google Group or contact me directly at alavrik@piqi.org or at @alavrik.
If you found a bug, please submit a GitHub issue.
Your feedback and participation is highly appreciated! If you find Piqi useful in some way or another, please drop me a line.
Project status
Piqi is a BETA software.
It has been tested on Linux and Mac OS X. Other Unix systems on major hardware architectures should generally work. For instance, Piqi is known to work on FreeBSD and Solaris/x86. On Windows, Piqi successfully builds under both MinGW and Cygwin. Refer to installation instructions for more details.
Check out the Roadmap page for the list of planned Piqi features.
