Gnu Ftp Mirror

broken image


The primary FTP site ftp.gnupg.org is mirrored at several sites. Please choose the nearest mirror to you. If you are seeking for web site mirrors, please consult the WWW mirror page. If you run a mirror of ftp.gnu.org and would like to have your site listed here, please contact gnu@gnu.org. If you would like to set one up.

Abstract

WP-MIRROR is a free utility for building mirrors of any desired setof Wikimedia Foundationwikis.

Which Wikis

The Wikimedia Foundation offers wikipedias in nearly 300languages. In addition, the WMF has several other projects(e.g. wikibooks,wiktionary,etc.) for a total of around 1000 wikis.

WP-MIRROR can build mirrors of any desired set of these wikis.

Why Build a Mirror

Gnu Ftp Mirrored

The main use cases for a mirror are these:
  • Development. If you are technically minded and need amirror with which you may conduct experiments;
  • Infrastructure. If you need redundancy, or need to servepages locally to minimize telecommunications traffic;
  • Offline browsing. If you need off-line access, perhaps forreasons of mobility, availability, and privacy; and
  • Research. If you need a mirror as a tool to assist yourresearch on the contents of any given wiki.

Key Features

WP-MIRROR builds a set of mirrors:

  • Appearance. A wiki page rendered by a mirror looks verysimilar to the same page rendered by the WMF servers;
  • Behavior. A wiki page rendered by a mirror behavesalmost the same (e.g. edit, search, user account creation, betafeatures); and
  • Completeness. Builds a complete mirror with original sizeimages.

WP-MIRROR is easy:

  • Easy to install. Available as a DEB package, and isavailable from a Debian package repository;
  • Easy to configure. The user may select any desired set ofwikis by editing just one line in a configuration file;
  • Easy to use. Sets up virtual hosts such as http://simple.wikipedia.site/,http://simple.wiktionary.site/,andhttp://www.wikidata.site/, onefor each wiki in the set, which the user may access with a webbrowser; and
  • Free software. Software is released under the GNU General Public License(GPLv3); and
  • Free documentation. Documentation is released under theGNU Free Documentation License,Version 1.3.

Out-of-the-box Experience

WP-MIRROR, by default, builds the following set of mirrors:

  • simple wikipedia,
  • simple wiktionary, and
  • wikidata wiki,

where Simple English means shorter sentences, and Wikidata is acentralized collection of facts usable by all other wikis (e.g. topopulate infoboxes).

The default works out-of-the-box with no userconfiguration. It should build in 200ks (two days), occupy 150G ofdisk space, be served locally by virtual hosts:http://simple.wikipedia.site/,http://simple.wiktionary.site/,andhttp://www.wikidata.site/, andupdate automatically every week.

The default should be suitable for anyone wholearned English as a secondlanguage (ESL).

Top Ten Wikipedias

The top ten wikipedias are the: en, de, nl, fr, it, es, ru, sv, pl, and ja wikipedias. BecauseWP-MIRROR uses original size media files, the top ten are too large tofit on a laptop with a single 500G disk, unless the user does not needthe images (and this is configurable).The en wikipedia is the mostdemanding case. It should build in 1Ms (twelve days), occupy 3T ofdisk space, be served locally by a virtual hosthttp://en.wikipedia.site/, andupdate automatically every month.

Most features are configurable, either through command-lineoptions, or via a configuration file(/etc/wp-mirror/local.conf).

Process

WP-MIRROR is non-interactive and normally runs in background as aweekly cron job, updating the mirror whenever the Wikimedia Foundationposts new dump files.

WP-MIRROR maintains the state of the mirror in a transactionaldatabase (InnoDB which is the ACID compliant storage engine forMySQL). There are three advantages to this:

  • Checkpointing. The state information is Durable (the `D'in ACID). When WP-MIRROR is interrupted (e.g. user closes laptop,power fails, cat walks across keyboard) the state information servesas a checkpoint. When WP-MIRROR is next started, it picks up where itleft off.
  • Concurrency. Multiple instances of WP-MIRROR can runconcurrently. That is to say, each instance of WP-MIRROR is Isolated(the `I' in ACID) from every other instance. The concurrency featureis intended for desktop use when one is mirroring any of the top tenwikipedias.
  • Monitoring. WP-MIRROR can also be run in monitor mode(concurrently with instances that are building mirrors). Instancesrunning in monitor mode display the state of each wikipedia. If asuitable windowing system is present, progress bars are rendered usinggraphics in a separate window, and otherwise using ASCII characters ina console (see figures below).

WP-MIRROR is designed for robustness. WP-MIRROR asserts hardwareand software prerequisites, skips over unparsable pages and bad filenames, waits for internet access when needed, and exits gracefully ifdisk space runs low.

Downloading WP-MIRROR

WP-MIRROR can be found on the main GNU server:http://download.savannah.gnu.org/releases/wp-mirror/(via HTTP).

Documentation

Documentation for WP-MIRROR is availableonline. The WP-MIRROR Reference Manual is available in PDF format.If you install from a package, the documentation will be registeredautomatically with `doc-base' and readily found using `dhelp' or`dwww'.

You may also find more information about WP-MIRROR by runninginfo wp-mirrororman wp-mirror,or by looking at/usr/share/doc/wp-mirror/,/usr/local/doc/wp-mirror/,or similar directories on your system. A brief summary is available byrunning wp-mirror --help.

Dependencies

WP-MIRROR has numerous dependences including: apache2, ImageMagick, MediaWiki, and MySQL. For this reason, it is easiest for theuser to install WP-MIRROR from a package.

WP-MIRROR 0.7 is available as a DEB package. It works`out-of-the-box' on Debian GNU/Linux 7.4 (wheezy) with backports andUbuntu 14.04 LTS (trusty). Porting to other distributions may beconsidered for a future release.

WP-MIRROR 0.6 is available as a DEB package. It works`out-of-the-box' on Debian GNU/Linux 7.0 (wheezy) and Ubuntu 12.10(quantal).

WP-MIRROR 0.5 is available as a DEB package. It works`out-of-the-box' on Debian GNU/Linux 7.0 (wheezy) and Ubuntu 12.10(quantal).

WP-MIRROR 0.4 is available as a DEB package. It works`out-of-the-box' on Debian GNU/Linux 7.0 (wheezy).

WP-MIRROR 0.3 and earlier versions, were developed on a PC with theDebian GNU/Linux 6.0 (squeeze) distribution installed. Userconfiguration of dependencies is required.

There are no plans to backport WP-MIRROR to earlierdistributions.

Installation

Debian GNU/Linux 7.4 (wheezy)

Method 1: Install from Debian package repository

1.1) Import the author's GPG public key into your root-shell's GPGkeyring, and into your APT trusted keyring:

1.2) Edit /etc/apt/sources.list by appending the `wheezy-backports'and the `debian-wpmirror' package repositories, like so:

If you are building your mirror on an IPv6 only network, then thelast line of /etc/apt/sources.list should instead read:

1.3) Upgrade your Debian distribution:

1.4) Install WP-MIRROR and its dependencies:

1.5) Run:

WP-MIRROR `just works'. Configuration is entirely automated; andthat includes configuration of dependencies such as `apache2',`MediaWiki', and `MySQL'.

Method 2: Download and install DEB packages

2.1) Releases are foundat http://download.savannah.gnu.org/releases/wp-mirror/. Select themost recent DEB packages, and install them in the following order:

2.2) Run:

Ubuntu 14.04 LTS (trusty)

Method 1: Install from Debian package repository

1.1) Import the author's GPG public key into your root-shell's GPGkeyring, and into your APT trusted keyring:

1.2) Edit /etc/apt/sources.list by appending the `debian-wpmirror'package repositories, like so:

If you are building your mirror on an IPv6 only network, then thelast line of /etc/apt/sources.list should instead read:

Gnu Ftp Mirror

1.3) Upgrade your Debian distribution:

Gnu Ftp Mirrors

1.4) Install WP-MIRROR and its dependencies:

1.5) Run:

WP-MIRROR `just works'. Configuration is entirely automated; andthat includes configuration of dependencies such as `apache2',`MediaWiki', and `MySQL'.

Debian GNU/Linux 6.0 (squeeze)

Mailing lists

WP-MIRROR has the following mailing lists:

Gnu ftp mirroring

1.3) Upgrade your Debian distribution:

Gnu Ftp Mirrors

1.4) Install WP-MIRROR and its dependencies:

1.5) Run:

WP-MIRROR `just works'. Configuration is entirely automated; andthat includes configuration of dependencies such as `apache2',`MediaWiki', and `MySQL'.

Debian GNU/Linux 6.0 (squeeze)

Mailing lists

WP-MIRROR has the following mailing lists:

  • wp-mirror-announceis used to announce releases
  • wp-mirror-develis a closed list for developers and testers.
  • wp-mirror-listis used to discuss most aspects of WP-MIRROR (e.g. feature requestsand bug reports).

Getting involved

Development of WP-MIRROR, and GNU in general, is a volunteereffort, and you can contribute. For information, please read How to help GNU. If you'd like toget involved, it's a good idea to join the discussion mailing list(see above).

Gnu Ftp Mirroring

Test releases
Trying the latest test release (when available) is alwaysappreciated. Test releases ofWP-MIRRORcan be found athttp://download.savannah.gnu.org/releases/wp-mirror/(via HTTP).
Development
For development sources, issue trackers, and otherinformation, please see theWP-MIRRORproject pageat savannah.gnu.org.
Translating WP-MIRROR
To translateWP-MIRROR'smessages into other languages, please see the Translation Projectpage forWP-MIRROR.If you have a new translation of the message strings,or updates to the existing strings, please have the changes made in thisrepository. Only translations from this site will be incorporated intoWP-MIRROR.For more information, see the TranslationProject.
Maintainer
WP-MIRROR is currently being maintained by Dr. Kent L. Miller. Please use the mailing lists for contact.

Gnu Ftp Mirror

Licensing

WP-MIRROR is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the FreeSoftware Foundation; either version 3 of the License, or (at youroption) any later version.





broken image