The Piqi Project

Really happy to have these out. Finally!

    piqi v0.6.6
    piqi-ocaml v0.7.0
    piqi-erlang v0.7.1
    piqi-rpc v0.7.1

Changelogs:

    https://github.com/alavrik/piqi/blob/master/CHANGES
    https://github.com/alavrik/piqi-ocaml/blob/master/CHANGES    
    https://github.com/alavrik/piqi-erlang/blob/master/CHANGES
    https://github.com/alavrik/piqi-rpc/blob/master/CHANGES

Updated roadmap:

    https://piqi.org/roadmap/

Here are some details.

1. piqi-ocaml turned into a separate project

One of the highlights of this release is separation of piqi-ocaml from
piqi. This completes the breaking of what used to be a monolithic
"piqi" repository into individual repos and, more importantly,
sub-projects.

In practice, it means a lot more independent development and releases
for all of them.

Piqi compiler for OCaml (now named piqic-ocaml) was rewritten on top
of the "piqi compile" interfaces. Together with piqic-erlang, which
was also improved as a result of this exercise, they make a canonical
example of how to write piqi backends.

There are a lot of improvements overall and some minor
incompatibilities in piqic-ocaml interfaces. See the changelog for
details.

2. piqi cleanup

No longer being held by piqi-ocaml, piqi sources layout has undergone
a major cleanup. Extremely pleased with the outcome:

- 3x faster build
- old multi-stage bootstrapping dropped in favor of a new clean system
  which is much easier to understand and use
- windows build overhaul -- now builds on modern OCaml for Windows out
  of the box
- support for cross-compiled win32 build
- dropped support for OCaml 3.11 and remove dependency on most Camlp4
  extensions

Overall, the core project has never been cleaner. For those who
considers hacking on it, this becomes very approachable now.

3. "Piqi is a universal schema language"

https://piqi.org now features a simplified and streamlined description
and project overview.

Check out the updated homepage and rationale
(https://piqi.org/rationale/).

4. Enhanced DSL capabilities

Perhaps all Piqi users know that .piqi syntax is based on the Piq
language. In a sense, Piqi is a declarative domain-specific language
that normally represented in Piq notation and defines itself via a
Piqi self-spec.

This combination turned out to be extremely powerful and flexible for
building other declarative DSLs. I've been it using quite successfully
for many small DSLs -- from config files and command-line interfaces
to SQL-like query languages.

The languages typically look nice, and, what's really cool, it takes
no time to build one. All you need is to describe your language in a
.piqi spec and Piqi takes care of both Piq-based concrete syntax and
automatically turning it into abstract syntax of your choice (Protocol
Buffers, JSON, XML and, of course, direct language representation).

A set of new features introduced in this release makes it even nicer:

- optionally omitting square brackets around record representation (so
  called frameless Piq input/output)
- optional quotes around single-word string literals (internally known
  as relaxed parsing)
- aliases for field and option labels that allow use of shorter names
  in Piq notation
- precise control over which syntax elements are allowed as positional
  elements and which should be always labeled

5. OCaml and Erlang: preserving unknown Protocol Buffers fields

A new --gen-preserve-unknown-fields flag was introduced for
piqic-ocaml and piqic-erlang. When specified, unrecognized Protocol
Buffers record fields are automatically captured during parsing and
get written back during serialization.

This helps to maintain reversibility of the representation making
applications that use older definitions forward-compatible.

I'm pleased to announce the release of Piqi-erlang and Piqi-RPC v0.7.0!

There were two major groups of changes: rewriting the Piqi-erlang code generator
in Erlang and moving Piqi-RPC and Piqi-Erlang into their own repositories.

Both changes are extremely important and represent a major milestone. The
particular benefits include:

- From now on, Piqi-erlang doesn't depend on the core Piqi project development
  and release cycles. Before that, it was pretty much impossible to release the
  three projects (Piqi, Piqi-erlang and Piqi-RPC) independently.

- Packaging and using Piqi-RPC and Piqi-erlang has become a lot simpler. Now
  they look and act almost like any other Erlang library.

- The fact that Piqi-erlang code generation is written in Erlang opens up a
  clear path for contributions from Erlang users. No OCaml knowledge or even
  OCaml installation is needed.

To simplify things even further, separate release git branches are gone and
everything is now in "master".

There was only one backward-incompatible change. The "piqic-erlang-ext" compiler
and therefore *_piqi_ext.erl go away. "piqic-erlang" now generates serializers
and deserializers for multi-format converters as gen_/2, /3 and
parse_/2, /3 functions) directly in *_piqi.erl. Migrating to the new
version should be as easy as changing all _piqi_ext:* function calls to call
_piqi:* instead.

Some other functional changes:

- "piqic-erlang --gen-defaults" option is deprecated. "piqic-erlang" now
  generates _piqi:default_X/1 without asking
- generate default values for Erlang record fields if they are defined in the
  original Piqi spec
- add a new "erlang-default" alias property; when specified, it overrides the
  Piqi-default value for the type in the generated "*_piqi:default_/0"
  functions

Along with the new Piqi-Erlang and Piqi-RPC versions, we are also releasing Piqi
v0.6.5. This is mostly a maintenance release -- same as the four previous Piqi
releases (v0.6.1 -- v0.6.4). Among other things it includes:

- a fix for incorrect parsing of large (> 2^63 - 1) unsigned decimal literals in
  Piq, JSON and XML
- a new "json-omit-missing" field property that controls, on a per-field basis,
  whether to exclude "null" and "[]" fields from JSON output
- better piqic tooling support, namely inclusion of the original .piqi file name
  in the compiled Piqi spec

We have also made a significant progress in non-functional areas such as:

- The original documentation for Piqi, Piqi-RPC and Piqi-erlang is now a part of
  source code tree. You can find it under the doc/ sub-subdirectory of
  correspondent repos and format it any way you like.
- Continuous builds and tests(!) through Travis-CI. Some new tests were added
  and interfaces to run the tests were automated.
- Initial Debian and RPM packaging

Overall, this was a major effort and it wouldn't have been possible without
support of many Piqi users. Thank you everyone for your enormous support,
contributions and patience.

Special thanks goes to Motiejus Jakštys for bug fixes and numerous other
contributions in the areas of OS packaging, documentation building scripts and
Travis-CI integration.

I would also like to thank Dennis Docter who troubleshooted an Erlang name
normalization problem in the new "piqic-erlang" implementation and contributed a
solution.

Links to the changelogs and updated roadmap:

https://github.com/alavrik/piqi/blob/master/CHANGES
https://github.com/alavrik/piqi-erlang/blob/master/CHANGES
https://github.com/alavrik/piqi-rpc/blob/master/CHANGES
https://piqi.org/roadmap/

I am proud to announce that Piqi version 0.6.0 is released!

This is the second major Piqi release and the second biggest
milestone in the history of the project.

Release highlights
------------------

1.  With this release, the Piqi language self-definition [1]
    became stable. Why is it a big deal? There are several
    implications.

    - Any .piqi module (including self-definition itself!) can be
      automatically and reliably converted to and from any
      supported portable data formats. It means JSON, XML and
      Protocol Buffers supported today, but also any other format
      that Piqi may support in the future.

    - In practical terms, it means that programs written in any
      language can consume and manipulate type definitions from
      .piqi modules. Also, because external representation of
      Piqi modules is not going to change in a backward
      incompatible way, applications built around Piqi type
      definitions won't be forced to change often (if at all).

    [1] https://piqi.org/self-definition/#piqi_piqi_source

2.  Support for untyped JSON and XML and other changes in
    encoding handling.

    - As a new experimental feature, it is possible to represent
      untyped JSON and XML as values of `piqi-any` type.
    - generalized and improved encodings schemes for top-level
      JSON and XML values
    - improved documentation

3.  Build and distribution improvements

    - conventional ./configure script
    - for OCaml users, Piqi can be now installed via
      OPAM package [2] (OPAM is a new package manager for OCaml)
    - greatly simplified installation from source code on Mac
      (thanks to OPAM)
    - Piqi documentation is now a part of the main Piqi GitHub
      repository [3]

    [2] https://github.com/ocaml/opam-repository/tree/master/packages/piqi.0.6.0
    [3] https://github.com/alavrik/piqi/tree/master/doc

Backward incompatible changes
-----------------------------

  - (piqi) Significant incompatibilities in external Piqi modules
    representation which means that older applications that rely
    on it won't work with the new version (e.g. older versions of
    "piqi call" command-line piqi-rpc client)
  - (piqi) Remove support for "piq-word" and "piq-text" built-in
    types and provide an alternative way to control Piq
    representation of string values using a new "piq-format"
    property
  - (piqi) Rename "piq-any" built-in type to "piqi-any"
  - (xml) always use  as a top-level XML element instead
    of tag derived from specific typename
  - (json) top-level values of primitive types and enum constants
    encoded in JSON as {"value": ...} instead of {"_": ...}
  - (tools) use $PIQI_PATH environment variable for Piqi module
    search paths instead of $PIQI_DIR

Full list of changes: https://github.com/alavrik/piqi/blob/master/CHANGES
Updated roadmap: https://piqi.org/roadmap/
Downloads: https://piqi.org/downloads/

For those Piqi-erlang and Piqi-rpc users who have upgraded or consider upgrading to the latest v0.5.7 stable release, please use v0.5.7-sp1 instead.

This service pack release fixes a problem with OTP release generation.

Piqi-0.5.7 is finally released!

Release highlights:

  - Better compatibility with Google Protocol Buffers

    The new version includes many fixes in "piqi to-proto" and also follows
    Protobuf behavior in handling more than one entriy for required and optional
    fields during binary deserialization

  - Improved upper-bound of record parsing time for Erlang, OCaml and Piqi
    tools. Essentially, it has improved from O(N^2) to O(N), where N is the
    number of fields.

  - Configurable serialization options for Erlang, OCaml, Piqi-RPC and Piqi
    tools:

    - pretty-print JSON and XML output
    - omit fields with 'null' values from JSON output
    - treat unknown and duplicate fields as errors when parsing JSON or XML
      input

  - Piqi-RPC fixes and improvements including the following:

    - Allow callbacks to modify Webmachine Request record; for example, this can
      be used for setting extra response headers, cookies, etc.
    - Optionally, do not include Erlang stracktraces in HTTP 500 responses when
      request handler crashes
    - Fix HTTP GET returning internal error when Piqi imports are used

  The next set of new features become especially useful for working with
  third-party Piqi or Protocol Buffers definitions which, for example, may not
  define necessary OCaml- or Erlang-specific properties in the first place.

  Now, it is possible to take any set of Piqi modules and write custom
  extensions for them without modifying the original files. Moreover, extensions
  can be loaded automatically for all recursively included and imported Piqi
  modules.

  - Add support for universal extensions

    Previously, Piqi extensions could be applied only to type definitions:
    records, variants, enums, lists and aliases.

    Starting from this version, they can be applied to all Piqi entries: type
    definitions, fields, options, functions, function parameters and imports.

  - Automatic inclusion of Piqi extension modules

    Extension module is a Piqi module that has a second extension in its file
    name. For example, "m.ocaml.piqi" is an extension module for a regular Piqi
    module "m.piqi".

    All operations applicable to regular Piqi modules are also supported for
    extension modules. The difference is that extension modules can be included
    automatically in the modules which they extend.

    For instance, "piqic ocaml" and "piqic erlang" try to automatically include
    .ocaml.piqi and .erlang.piqi respectively for each loaded module
    .piqi

Backward incompatible changes:

  - Change in Piqi-JSON mapping -- applicable to OCaml, Erlang serialization and
    "piqi convert" command:

    Piqi flags, i.e. fields and options without associated values, were
    previously encoded in JSON as

            "field_name" : null

    Starting from this version, their representation changes to

            "field_name" : true

    This is consistent with how flags are represented in binary format. It is
    also more reliable this way.

Full list of changes: https://github.com/alavrik/piqi/blob/master/CHANGES

Updated roadmap: https://piqi.org/roadmap/

Downloads: https://piqi.org/downloads/

This is the first Piqi release where most changes were driven by feedback from Piqi users. I have been receiving all kinds of it: bug reports, scalability and performance concerns, minor annoyances and usability problems, platform support and fairly major feature requests. Thanks to all Piqi users for the feedback and for staying involved!

This release includes many improvements as well as new features in various Piqi components.

First of all, Piqi now supports “packed” binary encoding for primitive numeric types. With this last piece, Piqi implements all Google Protocol Buffers features and becomes fully compatible with it.

For OCaml users, there’s one new big feature: multi-format data serialization. In addition to Protocol Buffers, support has been added for serializing OCaml values in JSON, XML and Piq formats. Check some examples here: https://github.com/alavrik/piqi/blob/master/examples/ocaml/io_json_xml_pb.ml

Other OCaml-related highlights include:

• An option to represent repeated fields and lists as OCaml arrays (by default they are represented as lists).
• Serialization of arbitrary OCaml types using a new extremely simple and flexible mechanism by mapping them to Piqi types. This mechanism can be used, for example, for serializing OCaml’s chars as integers or bigints as decimal strings. There are some examples at: https://github.com/alavrik/piqi/tree/master/examples/ocaml-custom-types/

Starting from this release, Piqi also builds on Windows using MinGW and Cygwin toolchains.

The full list of changes is available here: https://github.com/alavrik/piqi/blob/master/CHANGES

Downloads: https://piqi.org/downloads/

With this release Piqi graduates from alpha/prototype phase and enters Beta.

It has been almost 4 months since the previous release and plenty of new useful functionality has been added. Most of new functionality is related to Erlang. There are also some minor fixes and changes for OCaml.

This release introduces some backward incompatibilities with the previous release so make sure you check the ChangeLog before upgrading.

Release highlights:

• Multi-format data serialization for Erlang (JSON, XML, Protocol Buffers)
• Piqi-RPC — an RPC-over-HTTP system for Erlang
• Support for XML encoding
• Function definitions in the Piqi language
• Piqi getopt — type-based command-line arguments parsing

Full list of changes: http://github.com/alavrik/piqi/blob/master/CHANGES

Updated roadmap: https://piqi.org/roadmap/

Downloads: https://piqi.org/downloads/

There are two new major features in this release. Although not being immediately useful, they set an important foundation for future development.

The first one is the ability to serialize Piqi modules into the binary format in a way that ensures compatibility with future extensions of the Piqi language.

This is achieved by automatically tagging fields of the Piqi language data structures with hash-codes that are computed from field names. I learned this method from another powerful data serialization system for OCaml — ATD/biniou. The desired effect could also be achieved by assigning unique integer codes for each field manually, as it is done in Google Protocol Buffers, but the manual approach is less elegant and flexible for this use-case.

Another major feature of this release is the ability to embed type definitions in the data streams represented in Piq, JSON and the binary formats. This allows to produce self-describing data streams that include type definitions for the data contained in the stream. A new --embed-piqi flag for piqi convert command will do it automatically. Check the updated Piq documentation and Examples page to see how embedded Piqi modules are represented in Piq.

Release notes:

Now it is possible to embed Piqi type definitions into Piq streams to produce
self-describing data bundles. Piq streams with embedded type definitions can
be represented in Piq, Piq-JSON and Wire formats.

Piqi tools:

  - New "piqi convert --embed-piqi" flag for embedding type information into
    serialized Piq streams.
  - Use hash-based wire codes for Piqi self-specification instead of
    automatically-assigned ordinals to ensure forward and backward compatibility
    of serialized .piqi specs with regard to future Piqi language extensions.
  - Many internal improvements in Piqi compiler compiler, Piqi library
    initialization and serialization of .piqi specs.

piqic:

  - (ocaml, erlang) Generating more optimal code for variant parsing.

Bug fixes:

  - Fixed an integer overflow bug in piqirun-ocaml library on 32-bit platform.
  - Fixed missing location information in errors or warnings while converting
    from JSON.
  - Fixed incorrect field ordering during Piq input/output when Piqi
    specification explicitly defines non-ordinal wire codes.

This release introduces Piqi-light — a lightweight EBNF-like read-only notation for Piqi data definition language. It provides a compact way of displaying type definitions while omitting all non-significant properties that may be present in the original Piqi specification.

Unlike Piqi language, which is optimized for writing and flexibility, Piqi-light is optimized for readability.

More information about Piqi-light is available in Piqi documentation.

Also, Piqi examples, previously available only in the source code repository, are now uploaded and nicely displayed on the new Examples page. The new Piqi-light syntax for .piqi files is represented there as well.

Release notes:

Added initial support for Piqi-light -- lightweight EBNF-like syntax for Piqi
data definition language.

Piqi tools:

  - New "piqi light" command for printing .piqi in Piqi-light syntax.
  - Improved pretty-printing of .piqi by transforming output Piq AST. Mainly
    affects "piqi expand".
  - Added support for "-o" option to "piqi to-proto", "piqi of-proto" commands.
  - Improved floats reversibility in Piq (unified this part with JSON
    generator). Affects "proto convert -t piq".

piqic:

  - (ocaml) Simplified generated OCaml type definitions.

Bug fixes:

  - Timely reset location DB to allow garbage collection of objects. Affects
    "piqi convert" and "piqi pp".
  - Fixed several problems with "piqi expand".
  - Fixed printing order for "unknown field" warnings.

This release introduces Erlang support. I had been planning on implementing it for more than a year, and it finally happened!

Here are some Erlang support highlights.

• Generated Erlang type and record definitions as well as
  serialization/deserialization code are EEP8-compliant and provide all necessary
  type information for Dialyzer.
• Support for recursive record types.
• Support for Unicode strings which can be passed as string() (i.e. list of
  Unicode codepoints) or utf-8 binary().
• Support for custom Erlang identifiers. For example, if some identifier from
  existing Proto or Piqi specification conflicts with an Erlang keyword, it is
  possible to define a custom Erlang-specific identifier without changing the
  original one and breaking backward compatibility.
• Capitalized and “CamelCase” Piqi identifiers are automatically converted to
  uncapitalized and “camel_case” Erlang ids.
• Configurable prefixes for type and record identifiers (since Erlang doesn’t
  have any kind of namespace support for types and records, we need to prefix
  them to avoid conflicts with names defined in other modules).
• Tested with Erlang/OTP R13B04 and R14A.

Release notes:

Added Erlang support:

  - "piqic erlang" -- a new piqic backend for for generating Erlang type
    specifications and encoding/decoding functions.
  - "piqirun-erlang" -- runtime support library.

piqic:

  - (ocaml, erlang) Normalize Piqi identifiers by default, i.e. convert
    "CamlCase" identifiers to "caml-case".
  - (ocaml, erlang) Don't generate codecs and types for unused defintions from
    the boot module.

piqirun-ocaml:

  Backward incompatible changes:

  - Combined piqirun-ocaml library into one "piqirun" OCaml module (previously
    there were two: "piqirun_gen" and "piqirun_parse").
  - Improved piqirun-ocaml buffer initialization interface (unified it with
    piqirun-erlang).

  Other changes:

  - Ignore duplicate fields instead of producing an error.

Piqi tools:

  - Generate warnings instead of errors on duplicate fields when reading from
    Piq format.
  - Ignore duplicate fields when reading from wire format.

Platform support:

  - Now builds and runs on Mac OS X (fixed some build portability problems).