The Piqi Project

I am proud to announce that Piqi version 0.6.0 is released!

This is the second major Piqi release and the second biggest
milestone in the history of the project.

Release highlights
------------------

1.  With this release, the Piqi language self-definition [1]
    became stable. Why is it a big deal? There are several
    implications.

    - Any .piqi module (including self-definition itself!) can be
      automatically and reliably converted to and from any
      supported portable data formats. It means JSON, XML and
      Protocol Buffers supported today, but also any other format
      that Piqi may support in the future.

    - In practical terms, it means that programs written in any
      language can consume and manipulate type definitions from
      .piqi modules. Also, because external representation of
      Piqi modules is not going to change in a backward
      incompatible way, applications built around Piqi type
      definitions won't be forced to change often (if at all).

    [1] http://piqi.org/self-definition/#piqi_piqi_source

2.  Support for untyped JSON and XML and other changes in
    encoding handling.

    - As a new experimental feature, it is possible to represent
      untyped JSON and XML as values of `piqi-any` type.
    - generalized and improved encodings schemes for top-level
      JSON and XML values
    - improved documentation

3.  Build and distribution improvements

    - conventional ./configure script
    - for OCaml users, Piqi can be now installed via
      OPAM package [2] (OPAM is a new package manager for OCaml)
    - greatly simplified installation from source code on Mac
      (thanks to OPAM)
    - Piqi documentation is now a part of the main Piqi GitHub
      repository [3]

    [2] https://github.com/OCamlPro/opam-repository/tree/master/packages/piqi.0.6.0
    [3] https://github.com/alavrik/piqi/tree/dev/doc

Backward incompatible changes
-----------------------------

  - (piqi) Significant incompatibilities in external Piqi modules
    representation which means that older applications that rely
    on it won't work with the new version (e.g. older versions of
    "piqi call" command-line piqi-rpc client)
  - (piqi) Remove support for "piq-word" and "piq-text" built-in
    types and provide an alternative way to control Piq
    representation of string values using a new "piq-format"
    property
  - (piqi) Rename "piq-any" built-in type to "piqi-any"
  - (xml) always use  as a top-level XML element instead
    of tag derived from specific typename
  - (json) top-level values of primitive types and enum constants
    encoded in JSON as {"value": ...} instead of {"_": ...}
  - (tools) use $PIQI_PATH environment variable for Piqi module
    search paths instead of $PIQI_DIR

Full list of changes: https://github.com/alavrik/piqi/blob/master/CHANGES
Updated roadmap: http://piqi.org/roadmap/
Downloads: http://piqi.org/downloads/

For those Piqi-erlang and Piqi-rpc users who have upgraded or consider upgrading to the latest v0.5.7 stable release, please use v0.5.7-sp1 instead.

This service pack release fixes a problem with OTP release generation.

Piqi-0.5.7 is finally released!

Release highlights:

  - Better compatibility with Google Protocol Buffers

    The new version includes many fixes in "piqi to-proto" and also follows
    Protobuf behavior in handling more than one entriy for required and optional
    fields during binary deserialization

  - Improved upper-bound of record parsing time for Erlang, OCaml and Piqi
    tools. Essentially, it has improved from O(N^2) to O(N), where N is the
    number of fields.

  - Configurable serialization options for Erlang, OCaml, Piqi-RPC and Piqi
    tools:

    - pretty-print JSON and XML output
    - omit fields with 'null' values from JSON output
    - treat unknown and duplicate fields as errors when parsing JSON or XML
      input

  - Piqi-RPC fixes and improvements including the following:

    - Allow callbacks to modify Webmachine Request record; for example, this can
      be used for setting extra response headers, cookies, etc.
    - Optionally, do not include Erlang stracktraces in HTTP 500 responses when
      request handler crashes
    - Fix HTTP GET returning internal error when Piqi imports are used

  The next set of new features become especially useful for working with
  third-party Piqi or Protocol Buffers definitions which, for example, may not
  define necessary OCaml- or Erlang-specific properties in the first place.

  Now, it is possible to take any set of Piqi modules and write custom
  extensions for them without modifying the original files. Moreover, extensions
  can be loaded automatically for all recursively included and imported Piqi
  modules.

  - Add support for universal extensions

    Previously, Piqi extensions could be applied only to type definitions:
    records, variants, enums, lists and aliases.

    Starting from this version, they can be applied to all Piqi entries: type
    definitions, fields, options, functions, function parameters and imports.

  - Automatic inclusion of Piqi extension modules

    Extension module is a Piqi module that has a second extension in its file
    name. For example, "m.ocaml.piqi" is an extension module for a regular Piqi
    module "m.piqi".

    All operations applicable to regular Piqi modules are also supported for
    extension modules. The difference is that extension modules can be included
    automatically in the modules which they extend.

    For instance, "piqic ocaml" and "piqic erlang" try to automatically include
    .ocaml.piqi and .erlang.piqi respectively for each loaded module
    .piqi

Backward incompatible changes:

  - Change in Piqi-JSON mapping -- applicable to OCaml, Erlang serialization and
    "piqi convert" command:

    Piqi flags, i.e. fields and options without associated values, were
    previously encoded in JSON as

            "field_name" : null

    Starting from this version, their representation changes to

            "field_name" : true

    This is consistent with how flags are represented in binary format. It is
    also more reliable this way.

Full list of changes: https://github.com/alavrik/piqi/blob/master/CHANGES

Updated roadmap: http://piqi.org/roadmap/

Downloads: http://piqi.org/downloads/

This is the first Piqi release where most changes were driven by feedback from Piqi users. I have been receiving all kinds of it: bug reports, scalability and performance concerns, minor annoyances and usability problems, platform support and fairly major feature requests. Thanks to all Piqi users for the feedback and for staying involved!

This release includes many improvements as well as new features in various Piqi components.

First of all, Piqi now supports “packed” binary encoding for primitive numeric types. With this last piece, Piqi implements all Google Protocol Buffers features and becomes fully compatible with it.

For OCaml users, there’s one new big feature: multi-format data serialization. In addition to Protocol Buffers, support has been added for serializing OCaml values in JSON, XML and Piq formats. Check some examples here: https://github.com/alavrik/piqi/blob/dev/examples/ocaml/io_json_xml_pb.ml

Other OCaml-related highlights include:

• An option to represent repeated fields and lists as OCaml arrays (by default they are represented as lists).
• Serialization of arbitrary OCaml types using a new extremely simple and flexible mechanism by mapping them to Piqi types. This mechanism can be used, for example, for serializing OCaml’s chars as integers or bigints as decimal strings. There are some examples at: https://github.com/alavrik/piqi/tree/dev/examples/ocaml-custom-types/

Starting from this release, Piqi also builds on Windows using MinGW and Cygwin toolchains.

The full list of changes is available here: https://github.com/alavrik/piqi/blob/master/CHANGES

Downloads: http://piqi.org/downloads/

With this release Piqi graduates from alpha/prototype phase and enters Beta.

It has been almost 4 months since the previous release and plenty of new useful functionality has been added. Most of new functionality is related to Erlang. There are also some minor fixes and changes for OCaml.

This release introduces some backward incompatibilities with the previous release so make sure you check the ChangeLog before upgrading.

Release highlights:

• Multi-format data serialization for Erlang (JSON, XML, Protocol Buffers)
• Piqi-RPC — an RPC-over-HTTP system for Erlang
• Support for XML encoding
• Function definitions in the Piqi language
• Piqi getopt — type-based command-line arguments parsing

Full list of changes: http://github.com/alavrik/piqi/blob/master/CHANGES

Updated roadmap: http://piqi.org/roadmap/

Downloads: http://piqi.org/downloads/

There are two new major features in this release. Although not being immediately useful, they set an important foundation for future development.

The first one is the ability to serialize Piqi modules into the binary format in a way that ensures compatibility with future extensions of the Piqi language.

This is achieved by automatically tagging fields of the Piqi language data structures with hash-codes that are computed from field names. I learned this method from another powerful data serialization system for OCaml — ATD/biniou. The desired effect could also be achieved by assigning unique integer codes for each field manually, as it is done in Google Protocol Buffers, but the manual approach is less elegant and flexible for this use-case.

Another major feature of this release is the ability to embed type definitions in the data streams represented in Piq, JSON and the binary formats. This allows to produce self-describing data streams that include type definitions for the data contained in the stream. A new --embed-piqi flag for piqi convert command will do it automatically. Check the updated Piq documentation and Examples page to see how embedded Piqi modules are represented in Piq.

Release notes:

Now it is possible to embed Piqi type definitions into Piq streams to produce
self-describing data bundles. Piq streams with embedded type definitions can
be represented in Piq, Piq-JSON and Wire formats.

Piqi tools:

  - New "piqi convert --embed-piqi" flag for embedding type information into
    serialized Piq streams.
  - Use hash-based wire codes for Piqi self-specification instead of
    automatically-assigned ordinals to ensure forward and backward compatibility
    of serialized .piqi specs with regard to future Piqi language extensions.
  - Many internal improvements in Piqi compiler compiler, Piqi library
    initialization and serialization of .piqi specs.

piqic:

  - (ocaml, erlang) Generating more optimal code for variant parsing.

Bug fixes:

  - Fixed an integer overflow bug in piqirun-ocaml library on 32-bit platform.
  - Fixed missing location information in errors or warnings while converting
    from JSON.
  - Fixed incorrect field ordering during Piq input/output when Piqi
    specification explicitly defines non-ordinal wire codes.

This release introduces Piqi-light — a lightweight EBNF-like read-only notation for Piqi data definition language. It provides a compact way of displaying type definitions while omitting all non-significant properties that may be present in the original Piqi specification.

Unlike Piqi language, which is optimized for writing and flexibility, Piqi-light is optimized for readability.

More information about Piqi-light is available in Piqi documentation.

Also, Piqi examples, previously available only in the source code repository, are now uploaded and nicely displayed on the new Examples page. The new Piqi-light syntax for .piqi files is represented there as well.

Release notes:

Added initial support for Piqi-light -- lightweight EBNF-like syntax for Piqi
data definition language.

Piqi tools:

  - New "piqi light" command for printing .piqi in Piqi-light syntax.
  - Improved pretty-printing of .piqi by transforming output Piq AST. Mainly
    affects "piqi expand".
  - Added support for "-o" option to "piqi to-proto", "piqi of-proto" commands.
  - Improved floats reversibility in Piq (unified this part with JSON
    generator). Affects "proto convert -t piq".

piqic:

  - (ocaml) Simplified generated OCaml type definitions.

Bug fixes:

  - Timely reset location DB to allow garbage collection of objects. Affects
    "piqi convert" and "piqi pp".
  - Fixed several problems with "piqi expand".
  - Fixed printing order for "unknown field" warnings.

This release introduces Erlang support. I had been planning on implementing it for more than a year, and it finally happened!

Here are some Erlang support highlights.

• Generated Erlang type and record definitions as well as
  serialization/deserialization code are EEP8-compliant and provide all necessary
  type information for Dialyzer.
• Support for recursive record types.
• Support for Unicode strings which can be passed as string() (i.e. list of
  Unicode codepoints) or utf-8 binary().
• Support for custom Erlang identifiers. For example, if some identifier from
  existing Proto or Piqi specification conflicts with an Erlang keyword, it is
  possible to define a custom Erlang-specific identifier without changing the
  original one and breaking backward compatibility.
• Capitalized and “CamelCase” Piqi identifiers are automatically converted to
  uncapitalized and “camel_case” Erlang ids.
• Configurable prefixes for type and record identifiers (since Erlang doesn’t
  have any kind of namespace support for types and records, we need to prefix
  them to avoid conflicts with names defined in other modules).
• Tested with Erlang/OTP R13B04 and R14A.

Release notes:

Added Erlang support:

  - "piqic erlang" -- a new piqic backend for for generating Erlang type
    specifications and encoding/decoding functions.
  - "piqirun-erlang" -- runtime support library.

piqic:

  - (ocaml, erlang) Normalize Piqi identifiers by default, i.e. convert
    "CamlCase" identifiers to "caml-case".
  - (ocaml, erlang) Don't generate codecs and types for unused defintions from
    the boot module.

piqirun-ocaml:

  Backward incompatible changes:

  - Combined piqirun-ocaml library into one "piqirun" OCaml module (previously
    there were two: "piqirun_gen" and "piqirun_parse").
  - Improved piqirun-ocaml buffer initialization interface (unified it with
    piqirun-erlang).

  Other changes:

  - Ignore duplicate fields instead of producing an error.

Piqi tools:

  - Generate warnings instead of errors on duplicate fields when reading from
    Piq format.
  - Ignore duplicate fields when reading from wire format.

Platform support:

  - Now builds and runs on Mac OS X (fixed some build portability problems).

Major changes include support for JSON encoding and providing compatibility
with OCaml 3.12.0.

Piqi tools:

  - "piqi convert" command now supports two new encodings: "json" (plain JSON)
    and "piq-json" (JSON with Piq type annotation for top-level objects).
  - A new command: "piqi json-pp" for pretty-printing JSON files.

Piqi language:

  - Added support for "json-name" property for type definitions, fields and
    options. When specified, it will be used as a name for correspondent JSON
    field names instead of "name".
  - Renamed "ignore-field" top-level directive to "custom-field".

Other changes:

  - Provided compatibility with the latest OCaml release (3.12.0). Improved
    support for older OCaml versions >= 3.10.