Binary repository manager: Difference between revisions

Revision as of 16:17, 23 November 2015

Revision as of 20:36, 9 February 2016

Line 4:

== Introduction ==

Software development can be an extremely complex process<ref>{{cite web|last1=Biggert|first1=Johnny|title=SUSTAINABLE SOFTWARE DEVELOPMENT, PART 2: MANAGING COMPLEXITY|url=http://www.johnnybigert.se/blog/2012/02/sustainable-software-development-managing-complexity/|website=Developers Dilemma|publisher=Johnny Biggert|accessdate=11 January 2015}}</ref><ref>{{cite web|title=Managing Complexity|url=http://www.economist.com/node/3423238|website=The Economist|publisher=The Economist|accessdate=11 January 2015}}</ref> involving many developers, or teams of developers working on shared code bases, accessing the same build tools, downloading and using a shared set of binary resources, and deploying components into the same software product. To manage the source files used in software development, organizations will typically use [[revision control]]. The many source files used in software development are eventually built into the [[artifact (software development)|binary artifact]]s (also known as “binaries”) which constitute the components of a software product. In addition, in order to provide their functionality and feature set, software products may use many 3rd party artifacts downloaded from free open source repositories or purchased from commercial sources.<ref>{{cite web|title=Eighth Annual Future of Open Source Survey Finds OSS Powering New Technologies, Reaching New People, and Creating New Economics|url=https://www.blackducksoftware.com/news/releases/2014-future-open-source-survey-results-revealed|website=https://www.blackducksoftware.com|accessdate=25 February 2015}}</ref> Consequently, a software product may comprise tens, hundreds and even thousands of individual binary artifacts which must be managed in order to efficiently maintain a coherent and functional software product. This function of managing the binary artifacts is done by a binary repository manager. A binary repository manager can be thought of as being to binaries what revision control is to source files.

== What is a binary repository? ==

Line 47:

|}

== BinaryRelationship repository managers andto continuous integration ==

As part of the development lifecycle, source code is continuously being built into binary artifacts using [[continuous integration]]. This may interact with a binary repository manager much like a developer would by getting artifacts from the repositories and pushing builds there. Tight integration with CI servers enables the storage of important metadata such as:

* Which user triggered the build (whether manually or by committing to revision control)

* Which modules were built

* Which sources were used (commit id, revision, branch)

* Dependencies used

* Environment variables

* Packages installed

== What is a binary repository manager?<ref>{{cite web|title=What is a Binary Repository Manager|url=http://www.jfrog.com/binary-repository/|website=www.jfrog.com|publisher=JFrog Ltd.|accessdate=15 January 2015}}</ref> ==

In common day-to-day usage, the term “binary repository” is frequently used to refer to a “binary repository manager”, however, as the term suggests, one manages and the other is managed. A binary repository manager fulfills several functions for each step in the software development lifecycle.

Line 77 ⟶ 85:

A binary repository manager can enforce an organization’s development workflow by setting different permissions for each repository to only allow authorized users to promote artifacts from one repository to the next one in the workflow. For example, a release candidate must go through integration testing and QA before being made available to other teams. Using the binary repository manager, only authorized members of the QA team can promote the release candidate to the releases repository once it has passed the QA process. Then, the production systems can be configured to pull artifacts only from the releases repository.

==Universal Package Manager==

== Security and maintenance ==

The software and technology industry continues to change and grow, binary repository managers are no different. They are beginning to shift towards positioning as a universal package managers<ref> {{cite web |url= https://adtmag.com/articles/2015/09/08/jfrog-repository.aspx |title= JFrog Releases ‘Universal’ Artifact Repository |last=Waters |first= John K.| date= 8 September 2015|website=ADT Mag| publisher=Application Development Trends Magazine}}</ref>. These package managers aim to standardize the way enterprises treat all package types used in the [[software development process]]. They give users the ability to apply security and compliance metrics across all artifact types. Universal package managers have been referred to as being at the center of a [[DevOps]] toolchain<ref>{{cite web |url= http://www.codeproject.com/Reference/628210/An-Overview-of-the-NuGet-Ecosystem | title=An Overview of the NuGet Ecosystem | last=Decoster | first=Xavier | date= 18 August 2013| website=CodeProject.com}}</ref>.

== Key features of repository Managers==

=== Authentication and authorization ===

Key factors and features when consider the adoption of a package manager include: <ref>{{cite web |url= http://www.computerweekly.com/blogs/cwdn/2015/11/how-to-find-real-devops-look-for-binary-artifact-repository-control.html |title= How to find real DevOps, look for binary artifact repository control |last=Bridgewater | first=Adrian | date= 1 November 2015 |website=ComputerWeekly.com}}</ref>

Project source code is controlled by giving users access permissions as deemed necessary. In the same way, access to the resulting project binaries may also be controlled, and this can all be done by the binary repository manager. Moreover, repository managers can be integrated with other organization systems such as LDAP or Single Sign-on servers to simplify and centralize user management.

*'''Caching''' – Caching simply stores local copies of packages. This increases performance for slow internet connections by allowing the user to pull from the local repository instead of externally. Caching locally allows frequently used packages to be available even during times of external repository outages.

=== Purging policies ===

Binary repository managers can be used and configured to support organization purging policies to ensure reasonable disk space usage. For example, continuous integration servers may generate several snapshots of an artifact per day. Using the binary repository manager, an organization can configure purging based on number of snapshots or disk-space usage. Proxied repositories for third party artifacts may also be purged after not being used by any release for a specified time. For example, artifacts used during a proof-of-concept, may cease to be used once a product progresses to production.

*'''Retention policies''' - Repository managers can be used and configured to support organization purging policies to ensure reasonable disk space usage. Local repositories for third party artifacts may also be purged after not being used by any release for a specified time.

=== Managing third-party artifacts ===

Third party artifacts may be subject to approval processes due to licensing and legal issues. A binary repository manager can be used to allow usage of an artifact to development until the artifact is approved, and only then authorize publication to production repositories. Moreover, some binary repository managers (e.g., Artifactory) also include automatic license discovery and management and integration with license management software (e.g., BlackDuck Code Center).

*'''License filtering''' - Third party artifacts may be subject to approval processes due to licensing and legal issues. Package manager allow for restriction of only approved artifacts into deployment.

=== High availability ===

Since a binary repository manager maintains all the development dependencies, it is a central and usually mission-critical component in the organizational infrastructure. Any down-time of the binary repository manager can halt development with all the significant consequences to the organization. To overcome this risk, a binary repository manager can be installed with a High Availability Configuration to minimize the risk of down-time. This is achieved by having a redundant set of repository managers work against the same database and file storage. Through built-in processes of synchronization, each repository manager mirrors the other, so that no one repository manager can be a single point of failure.

*'''High availability''' - Since a binary repository manager maintains all the development dependencies, itits isvital ato centralalways andmaintain usuallyaccess mission-criticalto componentthese in the organizational infrastructureartifacts.. Any down-time of the binary repository manager can halt development with all the significant consequences to the organization. ToA overcomeHigh thisAvailability risk,instance aallows binaryan repositoryenterprise managerto canovercome bethe risk installedassociated with adowntime, Highthrough Availabilityautomatic Configuration to minimize the risk of down-timefailover. This is achieved by having a redundant set of repository managers work against the same database and file storage. ThroughMaintaining built-inenterprise processeswide ofstability synchronization,and eachperformance repositoryat managerall mirrorstimes the other, so that no one repository manager can be a single point of failure.

== Binary repository managers and continuous integration ==

As part of the development lifecycle, source code is continuously being built into binary artifacts using [[continuous integration]]. This may interact with a binary repository manager much like a developer would by getting artifacts from the repositories and pushing builds there. Tight integration with CI servers enables the storage of important metadata such as:

* Which user triggered the build (whether manually or by committing to revision control)

* Which modules were built

* Which sources were used (commit id, revision, branch)

* Dependencies used

* Environment variables

* Packages installed

*'''User restrictions''' - Repository managers can be integrated with other organization systems such as LDAP or Single Sign-on servers to simplify and centralize user management. This gives an enterprise granular control over who has access to vital software components.

This information can be used later for artifact scans or reports, audit and security checks and build traceability.

== Tools ==

== Binary repository manager software ==

Notable package mangers include: <ref>{{cite web |url= http://alternativeto.net/software/proget/ | title=ProGet |access-date=4 February 2016 | website=alternativeTo.net}}</ref>

Some software products that try to fulfill the requirements of a binary repository manager:

* JFrog's ArtiFactoryArtifactory

* Apache Archiva

* SonaTypeSonatype Nexus

* Inedo's [[ProGet]]

* Package Drone

Apart from the above, multiple repositories are targeted to publish binaries to external parties, often for typical platforms ([[Android (operating system)|Android]], [[iOS]], [[Rpm (red hat)|RPM]]s for [[Linux]], ...)