The program you write will depend on code that other people have written. This guide will teach you how to install, use and upgrade dependencies for your web application. It covers ruby, javascript and php.
After working through this guide you will be able to:
npm
gem
and bundler
composer
brew
, apt
or choco
The program you write will depend on code that other people have written. Some of this code has been packaged into operating systems or applications: Linux, Postgres, Apache, nginx, and so on.
But you also use smaller pieces of code that you include in your source code or link to your code after compiling. These dependencies are called libraries, packages or gems in different programming languages.
We will discuss six different systems:
apt
a package manager for Linux (used in debian, ubuntu, and several other distributions)brew
a package manager for mac oschoco
a package manager for windowsgem
and bundler
for rubynpm
for javascript and node.jscomposer
for phpWhile this chapter is about using dependencies, you might also consider what goes into creating a good dependency: how do I write code that can be reused? How do I split up a larger problem into smaller pieces that can be reused?
Therese are questions of software design and software architecture. They go back a long time (e.g. D.L. Parnas. (1972). On the criteria to be used in decomposing systems into modules. Communications of the ACM, 15(12), 1053–1058. doi: 10.1007/978-3-642-48354-7_20, discussed in morning papers).
The site libraries.io collects information for many package management systems, it's a central place to look for software to use.
Why do we need to look at apt
, brew
or choco
? Isn't the
package manager for my main programing language enough?
In many cases using php-only or ruby-only libraries will be enough. But sometimes the libraries we use in our dynamic languages use other code originally written in c or c++.
The image processing library imagemagick
is a good example. If you want to use the ruby gem rmagick or the npm package easyimage to edit images you will that they deplend on the library "ImageMagick"
which is writen in C.
You do this using apt or brew or the windows installer with dynamic link libraries.
sudo apt-get install imagemagick libmagickwand-dev # ubuntu, wls brew install imagemagick # macos
The result of installing imagemagick with brew is the file libMagickCore-6.Q16.2.dylib
and two other libraries. These are libraries that can be used by multiple programs on your computer when they are dynamically linked to those programs.
When installing the ruby gem with gem install rmagick
you get the following output:
Building native extensions. This could take a while...
Sometimes you need to set include paths when installing gem, e.g.:
> gem install eventmachine ... error messages ... In file included from binder.cpp:20: ./project.h:116:10: fatal error: 'openssl/ssl.h' file not found #include <openssl/ssl.h> ^ > brew install openssl > gem install eventmachine -- --with-cppflags=-I/usr/local/opt/openssl/include
To enable easy installation of a dependency with just one command we need a central database that contains the information on all available packages. These databases typically also offer a web page with search functionality:
/etc/apt/sources.list
apt-get install imagemagick
brew install imagemagick
choco install imagemagick
gem install rmagick
npm install imagemagick
composer install imagemagick
- does not work, needs PECLAll the package managers distinguish between installing globally - for all users and projects on a computer and installing locally. This might mean installing in a way that only one user can use, or it might mean installing into a projects directory.
When you deploy your project to a production server you again face this question: should the dependencies be installed globally or locally?
Downloading and using software is always dangerous. With a package management system a lot of downloads happen automatically. So we want to make sure the software we download is really what it is supposed to be.
A package management system can offer different methods of making it more secure:
For a big web project you will be using a lot of dependencies. This will lead to two problems (here shown in a ruby projects with gems):
If this all goes horribly wrong you are in "dependency hell": you can't find the right versions of gems to make your code work again on a new machine or after an update.
Bundler is the name of the ruby tool that solves this problem for ruby.
Bundler is itself a gem, so you install it with gem install bundler
.
Beware: the command you will be using called bundle
, not bundler.
This is how it works: In every ruby project you write
a Gemfile
in which you define which gems and (to a certain degree) which versions you want.
When you run bundle install
bundler will:
Gemfile.lock
The lock-file contains a complete list of all the gems necessary for your project, and their version numbers. This also includes dependencies of you dependencies, that you don't even knew you were using! The version numbers are now locked down, and will not change.
When deploying to a new development machine or the production server,
you run bundle install
and the exact same versions are now installed.
All modern package managment systems offer a form of this solution, with
You can add packages by editing the Gemfile, or on the commandline with bundle add
and
the name of the gem.
When definig your wishes for dependencies you can specify which versions should be used. But don't overdo it! The package manager does a good job picking versions. If you specify every version number by hand you are doing too much work. And you are potentially locking yourself in to old versions.
Some examples of the different ways of specifying version number and source in a ruby Gemfile
:
# Gemfile source 'https://rubygems.org' ruby '2.1.5' gem 'devise' gem 'rails', '4.2.5' gem 'uglifier', '>= 1.3.0' gem 'coffee-rails', '~> 4.1.0'
Giving an exact version number fixes that version, so only version 4.2.5 of rails will be allowed here. Using the "greater-or-equal-than" sign you can require any version greater or equal to 1.3.0. So 1.2.7 is forbidden, but 1.3.2 or 1.4.0 or 2.2.0 is allowed.
The arrow ~>
will only allow an increase in the
last (right most) number, so ~> 4.1.0
does allow 4.1.17
but not 4.2
.
This is called a pessimistic version constraint, read more about
it in the rubygem documentation.
There now is a common standard on how to use version numbers called semantic versioning. In short: Use three numbers for your version number: MAJOR dot MINOR dot PATCH. For example 3.1.0.
But not all projects adher to this standard yet.
Let's assume you list 12 dependencies in your project. This might result in 36 dependencies being installed in all. Now assume each dependency releases a new version once a year. You have to be prepared for updating several dependencies each month!
To update the versions you need to override the lock-file and allow a change in version numbers.
For each dependency system we discussed there are services out there that will tell you about new version or -- even more important -- new security updates for your dependencies.
Using a third party library will make you more productive today. But it could well harm you in the long run:
These examples were taken from a longer article that discusses ways to minimize the risk.
The Advanced Package Tool, or APT, handles the installation and removal of software on the Debian, Ubuntu and other Linux distributions. It installs both complete programs (like apache in the example below) and libraries (like libssl in the example below) that can be used to build programs.
apt install apache2 apt install libssl-dev
The software is downloaded from a file system (e.g. a DVD) or an FTP server.
Each Linux distribution keeps their own server.
You can search for software with apt search
$ apt search xml [...] xmlto/xenial 0.0.28-0.1 amd64 XML-to-any converter xmltoman/xenial,xenial 0.4-3 all simple XML to man converter xmltooling-schemas/xenial,xenial 1.5.6-2 all XML schemas for XMLTooling [...]
The software is installed into the usual folder used in UNIX (see also the Filesystem Hierarchy Standard (FHS):
/etc
for configuration files/bin
and /lib
for essential binaries and libraries/usr/bin
and /usr/lib
for non-essential (=most) binaries and librariesYou can find out what exactly got installed by running the dpkg
command:
$ dpkg -L libmagickcore /. /usr /usr/lib /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu/libMagickCore-6.Q16.so.2.0.0 /usr/lib/x86_64-linux-gnu/ImageMagick-6.8.9 /usr/lib/x86_64-linux-gnu/ImageMagick-6.8.9/modules-Q16 /usr/lib/x86_64-linux-gnu/ImageMagick-6.8.9/modules-Q16/filters /usr/lib/x86_64-linux-gnu/ImageMagick-6.8.9/modules-Q16/filters/analyze.so /usr/lib/x86_64-linux-gnu/ImageMagick-6.8.9/modules-Q16/filters/analyze.la [...]
As apt installs many types of software, there are many ways to use this software:
wget http://io9.com
service apache2 start
#include <PCSC/winscard.h>
and then link the libraryHomebrew is a package manager for Mac OS. It builds software from source, applying patches to make the software mac compatible.
It installs both complete programs (like postgresql in the example below) and libraries (like libyaml in the example below) that can be used to build programs.
brew install postgresql brew install libyaml
You can search for software with brew search
:
$ brew search xml [...] homebrew/php/php54-wbxml libxml++3 homebrew/php/php54-xmldiff libxml2 ✔ homebrew/php/php55-wbxml libxmlsec1 [...]
Homebrew is a git repository in /usr/local/Homebew
. It installs
software into /usr/local/Cellar
and then creates links from
/usr/local/bin
or /usr/local/lib
as appropriate.
You can find out what exactly got installed by running the brew list
command:
$ brew list libyaml /usr/local/Cellar/libyaml/0.1.7/include/yaml.h /usr/local/Cellar/libyaml/0.1.7/lib/libyaml-0.2.dylib /usr/local/Cellar/libyaml/0.1.7/lib/pkgconfig/yaml-0.1.pc /usr/local/Cellar/libyaml/0.1.7/lib/ (2 other files) $ brew list postgresql /usr/local/Cellar/postgresql/9.5.4_1/bin/clusterdb /usr/local/Cellar/postgresql/9.5.4_1/bin/createdb [...]
The executable programs in the last example can
be found as links in /usr/local/bin/
lrwxr-xr-x user clusterdb -> ../Cellar/postgresql/9.5.4_1/bin/clusterdb lrwxr-xr-x user createdb -> ../Cellar/postgresql/9.5.4_1/bin/createdb
As brew installs many types of software, there are many ways to use this software:
wget http://io9.com
brew services start postgresql
#include <yaml.h>
and then link the libraryLibraries for ruby are called gems and can be installed using the command lin tool gem
:
gem install pg
The gems are stored on a central server https://rubygems.org
.
You can use gem search
or the web interface to find gems.
$ gem search xml [...] klarna-xmlrpc (0.2.1) koara-xml (0.13.0) latexml-ruby (0.0.2) libxml-ext (0.4.2) [...]
Gems will normally be installed "globally" = in a way that they can be used by several users for several projects.
C:/RailsInstaller/Ruby2.2.0/lib/ruby/gems/2.2.0
on Windows/usr/lib/ruby/2.3.0/
on LinuxYou can find out where a gem is installed by using gem which
.
On many machines rvm
- the ruby version manager - is used to switch
between different versions of the programming language. Then each version
of the language has it's own set of gems, and the paths include the
word rvm
:
/usr/local/rvm/gems/ruby-2.2.3/gems/pg-0.18.4/lib/pg.rb
Unix, global rvm, ruby version 2.2.3, gem pg
for postgresql/Users/USERNAME/.rvm/rubies/ruby-2.3.0/gems/pg-0.18.4/lib/pg.rb
Windows, rvm installed for one user only, ruby version 2.3.0, gem pg
for postgresqlTo use gems in your ruby project you will write a Gemfile
where you
list the gems (and maybe the versions) that you need, e.g.:
source 'https://rubygems.org' gem 'activerecord' gem 'pg' gem 'mechanize'
To install all the gems and their dependencies run the command bundle
.
The bundler will find a constellation of dependencies and versions
that work well together, and write this information to Gemfile.lock
.
In a non-rails project you load the gems in the ruby files that need
them with require
:
require 'active_record'
In a rails project this is not necessary, the gems are loaded by rails.
When using git with ruby you add Gemfile
and Gemfile.lock
to the
repository.
Keeping up to date:
Libraries for javascript are called packages and can be installed using the command line tool npm
:
npm install should
You can search for packages using npm search
or the web search interface
at https://www.npmjs.com/.
$ npm search xml [...] muxml-cli Streaming XML parser and formatter =t1st3 mxmlc Node.js module to build ActionScript =meekgeek naive-request Very simple xmlhttp library for the… =kev_nz named-items-rss RSS feed generator. =jamierevans [...]
Packages will normally be installed "locally" in the sub-folder node_modules
.
If you want to install them globally add -g
to the command:
npm install -g grunt
To use packages in your javascript project you will write a package.json
file where you list the packages (and maybe the versions) that you need. To create
the first version of this file use npm init
. Then you can use the --save
option to install packages and at the same time add them to the file:
$ npm install --save should my_first_javascript_project@1.0.0 /Users/USERNAME/mfjsp └─┬ should@11.1.1 ├── should-equal@1.0.1 ├── should-format@3.0.2 ├── should-type@1.4.0 ├── should-type-adaptors@1.0.1 └── should-util@1.0.0 $ cat package.json { "name": "my_first_javascript_project", "version": "1.0.0", "description": "reimplementation of leftpad", "main": "index.js", "scripts": { "test": "echo \"Error: no test specified\" && exit 1" }, "author": "Maria Musterfrau", "license": "ISC", "dependencies": { "should": "^11.1.1" } }
In a serverside node.js project load the packages with require
. Before ECMAScript2015:
const http = require('http');
In ECMAScript2015:
import { http } from 'http';
To fix the exact versions of all dependencies, use
npm shrinkwrap
. This command creates a file npm-shrinkwrap.json
.
When using git with a node.js project you add packages.json
and npm-shrinkwrap.json
to the repository, and node_modules
to .gitignore
.
One package manager for php is composer
. It can be used to
install packages with composer require
:
$ composer require friendsofphp/php-cs-fixer Using version ^1.12 for friendsofphp/php-cs-fixer ./composer.json has been created Loading composer repositories with package information Updating dependencies (including require-dev) - Installing sebastian/diff (1.4.1) Downloading: 100% - Installing symfony/stopwatch (v3.1.6) Loading from cache [...] - Installing friendsofphp/php-cs-fixer (v1.12.3) Downloading: 100% Writing lock file Generating autoload files
It installs the software and it's dependencies into the subfolder vendor
. It also
generates a composer.json
and a composer.lock
file.
To install packages globally use global:
$ composer global require friendsofphp/php-cs-fixer
To use the packages in you php script you only need to
require one file: autoload.php
. Then you can start using
the classes loaded from the packages:
require __DIR__ . '/vendor/autoload.php'; $log = new Monolog\Logger('name');
When using git with a php project you add composer.json
and composer.lock
to the repository, and vendor
to .gitignore
.
A word of warning: there are two older levels of dependencies:
apt
for Linux, e.g. for Ubuntu and Debian
brew
for mac os brew.sh
choco
for windows chocolatey.org
gem
and bundler
for ruby rubygems.org and bundler.io
npm
for javascript and node.js npmjs.org
composer
for php getcomposer.org and packagist.org
/