Home / Erlware / Repository Reference

Why a Centralized Repository?

It's not really a centralized repository. Erlware is the community repository, it is a well known place to get OTP applications provided by the Erlang community. In fact, we expect individual developers and enterprises alike to provide a local repository for their own purposes. We document the repository layout so people can do just that. With Sinan. our build system, you can specify as many repositories as you like and it will search them in the order you specify. So there although Erlware is and should be a well known widely available repository it shouldn't be seen as a centralized solution.

Why Not Existing Repositories?

The repository you are probably thinking of is [CEAN][cean] which is owned by Process One and maintained by Mickaël Rémond and Christophe Romain. CEAN is actually a pretty interesting system with quit a large number of packages available. Unfortunately, it doesn't actually fit our needs for three main reasons.

Point #1 is one of the biggest problems for us. Based on our development experience we know that it is very important for an entity to be able to host their own repositories. This is especially true of companies that build proprietary software. They need to be able to host their internal code on their own repositories with no chance of that code leaking out of their firewalls. Its also important for individuals working on open source projects who's code may not be ready for distribution into a widely available repository. Perhaps they have a small private developer group that needs access to the code. In any case, We think that the best way to allow these use cases is to document the repository format well and let people host their own. CEAN plans to provide documentation for their backend at some point in the future. When that happens this issue may no longer be an issue.

Point #2 A very important feature of our build system is the ability to just 'do the right thing' with minimal user input during the build process. Like one of the popular web development frameworks out today we favor convention over configuration. We rely on a well defined structure to the project as well as the use of metadata. OTP provides that structure and metadata for us. Therefore we require that the applications we work with follow OTP standards. We also hope that this encourages the spread of OTP and OTP knowledge through out the Erlang community - one of Erlware's main goals. CEAN also requires an OTP layout though not *.app files. It doesn't expose this data in a programmatically useful way however.

Point #3 One of the big features of our build system is the ability to manage dependencies. This is a very nice feature that we need to support and we can't do that without easily accessible metadata on remotely available dependencies.

This is in no way meant as a criticism of CEAN. CEAN and our repository where created with very different goals in mind. CEAN was created as a CPAN like structure for Erlang where as our repository follows something more akin to Maven's repository model. Each has evolved to fill its own niche.

Admission into Erlware

If you want to submit your application or release (from now on applications and releases will be generically referred to as "packages") to Erlware for inclusion in the main repository; Congratulations! This is what Erlware is all about. When a package is submitted to Erlware it must meet certain standards in order to be included. That applications conform to these standards is ensured through our verification process. Unlike with a package that is stored in a local repository a package submitted to Erlware does not go straight into the repo, instead it is placed in a special staging area. From there community members work to ensure the application meets the minimum requirements. Requirements for the Apps That Constitute the Erlware Stable Repository

Goals and Requirements of the Repository Project

Requirements of the Repository

Repository Layout

The repository layout is straight forward. A developer familiar with debian and the rubygems projects should recognize a similar structure here. We believe that we have taken into account all of the important dimensions of source and binary projects. We have also worked to design in a metadata facility to the repository. To that end the repository looks as follows

<repo>/<erts-vsn>/<arch>/<lib|releases><package-name>/<package-vsn>/<package-name>.tar.gz

Repository Metadata

The repository metadata is under the same top level but layered slightly differently.

<repo>/<erts-vsn>/Meta/<package-name>/<package-vsn>/<tag files>

In this case, under the Meta directory is the name of the OTP application contained within. Finally there is the metadata named by <package-vsn>. The <package-vsn> should represent the version of the package this metadata applies to. The metadata itself is a *.app file. The .app file provides enough information for dependency management while remaining easily accessible.

Tag file(s) are simple files that give additional metadata about the package. One file that is almost always present is the license file. Another is the native file that indicates that this package contains native code. There is also the deprecated file that indicates that a package has been deprecated.

ERTS in The Repository

<repo>/<erts-vsn>/<arch>/erts.tar.gz

Under the erts directory is a tarball containing the latest erts version compiled for the architecture specified by arch. The version-list file is a list of versions of erts available. This file is in the same format as the one used in the application metadata area.

Getting Things Into The Repository

Faxien is the Erlware utility that will allow you to interact with an Erlware software repository. faxien fetchs packages from the remote repository and installs them on your local machine. Conversely faxien also publishes packages located on your local hard disk into a remote repository. For more information see the Faxien docs.

Specifying Additional Application Dependency Information

One thing that normal OTP application *.app lacks is the ability to specify versioned dependencies at the OTP Application level. Usually, this happens at the *.rel level. However, its hard to know what your dependencies are without going through a large amount of manual effort looking at *.rel files for various projects. It also makes it hard for a repository based on OTP applications as the unit of distribution, like ours. To that end we have added a small bit of addition information *.app files that constitute our metadata. That information is a versioneddependencies and an ertsvsn tuple. The OTP release handling structure ignores this information, but it is used by our dependency resolution infrastructure. An example of *.app file with this additional information follows.

{application, someapp,
 [{description, "Versioned Dependencies example"},
  {vsn, "0.1.0"},
  {erts_vsn, "5.5.3"},
  {modules, [sa_app,
             sa_sup,
             sa_worker,
             sa_worker_sup]},
  {registered, [tercio]},
  {applications, [kernel, stdlib, sasl]},
  {versioned_dependencies, [{kernel, "2.11.2"},
                            {stdlib, "1.14"},
                            {sasl, "2.1.4", gte}]},
  {mod, {sa_app, []}}]}.

Neither of these entries are required. If they are left out the dependency resolution scheme will happily use whatever is the latest available in the repository. However, if an app has a hard version requirement it is a good idea to specify these dependencies. In our experience this is usually not necessary.

Special notes: You may specialize the dependecy information by using a part of the version number or by adding gte no the tuple. Using part of a version number indicates that anything within that range is acceptable. For example, if you specify stdlib 1.14 then either stdlib 1.14.34 and 1.14.99 but not 1.15.0 are acceptable. gte works a bit similarly. If you wish to say that any version greater then or equal to sasl 2.1.4 is acceptable just add gte no the tuple.

Hosting Your Own Repository

We have tried to keep hosting your own repository very, very simple. To that end we settled on requiring a simple HTTP server for repository access. Because we support putting packages into the repository and pulling information about the repository down we also require that HTTP server to be dav enabled. The downside to this is that not every HTTP server supports dav and those that do tend to be a bit more complex then those that don't. Even with that being the case we believe that requiring a dav enable HTTP server is a simpler option then providing a custom repository application that repository hosters would need to install and run. This also reduces the maintenance burden on those of us that help out with erlware.

Two servers that we use that support dav: Apache with the mod_dav module and lighttpd. We tend to use lighttpd because its lightweight and performant. However, an dav enable server will do.