How to Set up nZEDb Usenet Indexer on Ubuntu 16.04/17.04

Previously we discussed how to install SABnzbd Usenet client on Ubuntu 16.04/17.04. This tutorial is going to show you how to set up your own nZEDb Usenet indexer on Ubuntu 16.04/17.04 so that you can find content on Usenet conveniently. nZEDb is a free and open-source (GPL-licensed) Usenet indexer. It’s a fork of NewzNab plus, which is closed-source and commercial Usenet indexer.

Why set up your own Usenet indexer? There are a number of reasons:

• It’s free of charge.
• You can download as many NZB files as you like.
• Unlimited API calls.
• Third-party Usenet indexers can disappear at any time.
• You don’t have to give your personal information to anyone when using your own Usenet indexer.

Prerequisites

In order to follow this tutorial, you will need a Ubuntu 16.04/17.04 computer running with LAMP or LEMP stack. If you haven’t already done so, please check out one of the following tutorials.

• How to install LAMP (Apache, MariaDB, PHP7) on Ubuntu 16.04
• How to install LEMP (Nginx, MariaDB, PHP7-FPM) on Ubuntu 16.04

Once LAMP or LEMP is installed, come back here and read on. You also need to have a Usenet account, the details of which will need to be entered in the nZEDb web installer. Now follow the instructions below to install nZEDb on Ubuntu 16.04/17.04.

Configuring MariaDB

Edit MariaDB server configuration file using nano text editor.

sudo nano /etc/mysql/mariadb.conf.d/50-server.cnf

Add the following lines in the [mysqld] section.

# configurations by nZEDb
innodb_file_per_table = 1
max_allowed_packet = 16M
group_concat_max_len = 8192

Save and close the file. Then restart MariaDB for the changes to take effect.

sudo systemctl restart mysql

Creating a Database and User in MariaDB

Log into MariaDB database server with the following command:

mysql -u root -p

Then create a database for nZEDb. This tutorial name the database nzedb. You can use whatever name you like.

create database nzedb;

Create the database user and grant this user all privileges on nzedb database. Replace your-password with your preferred password.

grant all privileges on nzedb.* to 'nzedb'@'localhost' identified by 'your-password';

Also grant file permission to this user.

grant file on *.* TO 'nzedb'@'localhost';

Flush privileges and exit.

flush privileges;

exit;

Telling AppArmor to Ignore MariaDB

AppArmor is a Linux security module enabled by default on Ubuntu. However, it prevents nZEDb from using the SQL’s LOAD DATA commands. We need to tell AppArmor to ignore MariaDB using the commands below.

sudo apt install apparmor-utils

sudo aa-complain /usr/sbin/mysqld

You might encounter the following error. It tells you that MariaDB isn’t restricted by AppArmor so just go to the next step.

ERROR: /etc/apparmor.d/usr.sbin.mysqld contains no profile

Installing Media Tools

Run the following command.

sudo apt-get install unrar p7zip-full mediainfo lame ffmpeg libav-tools

Next, execute the following commands to download, compile and install yEnc, which speeds up header and message processing during indexing.

cd ~
mkdir yenc
cd yenc
wget https://ayera.dl.sourceforge.net/project/yydecode/yydecode/0.2.10/yydecode-0.2.10.tar.gz
tar xzf yydecode-0.2.10.tar.gz
cd yydecode-0.2.10
sudo apt install gcc make
./configure
make
sudo make install
cd ../..
rm -rf ~/yenc

Install php-yenc extension for even faster header and message processing.

wget https://github.com/niel/php-yenc/releases/download/v1.3.0/php7.0-yenc_1.3.0_amd64.deb

sudo dpkg -i php7.0-yenc_1.3.0_amd64.deb

Installing nZEDb

Make a directory for nZEDb.

sudo mkdir /var/www/nZEDb/

Set web server user (www-data) as the owner and grant write permission to www-data group.

sudo chown www-data:www-data /var/www/nZEDb -R
sudo chmod g+w /var/www/nZEDb/ -R

Add your user account to www-data group.

sudo adduser username www-data

Log out and log back in for this change to take effect. Then run the following command to change the current login group to www-data.

newgrp www-data

We will use the git tool to clone nZEDb repository from Github. So install git on Ubuntu 16.04/17.04 with the following command.

sudo apt install git

Change directory to /var/www/.

cd /var/www/

Then clone nZEDb repository.

git clone https://github.com/nZEDb/nZEDb.git

Under /var/www/nZEDb/_install/ directory there’s a install script for nZEDb. We need to copy it to /var/www/.

sudo cp /var/www/nZEDb/_install/install.sh /var/www/

Add execute permission.

sudo chmod a+x /var/www/install.sh

Before running this script, we need to install some PHP extensions.

sudo apt install php7.0-mcrypt php-imagick php-pear php7.0-curl php7.0-gd php7.0-json php7.0-dev php7.0-gd php7.0-mbstring php7.0-xml

Now make sure you are in /var/www/ directory and run the install script.

cd /var/www/

./install.sh

During the installation, Composer will install all needed PHP dependencies and you will need to enter your sudo password to set permissions. Once the script finished, we need to set up the web server. You can use either Apache or Nginx.

Setting up Apache Virtual Host

Create a virtual host in /etc/apache2/sites-available/ directory for nZEDb.

sudo nano /etc/apache2/sites-available/nzedb.conf

Put the following text into the file. Replace 12.34.56.78 with your Ubuntu server’s IP address. You can also use a domain name.

<VirtualHost *:80>
    ServerAdmin webmaster@localhost
    ServerName 12.34.56.78
    DocumentRoot "/var/www/nZEDb/www"
    LogLevel warn
    ServerSignature Off
    ErrorLog /var/log/apache2/error.log
    <Directory "/var/www/nZEDb/www">
       Options FollowSymLinks
       AllowOverride All
       Require all granted
    </Directory>
    Alias /covers /var/www/nZEDb/resources/covers
</VirtualHost>

Save and close the file. Then disable default virtual host and enable the new virtual host.

sudo a2dissite 000-default

sudo a2ensite nzedb.conf

Enable rewrite module.

sudo a2enmod rewrite

Reload Apache for the changes to take effect.

sudo systemctl reload apache2

Now visit the following URL to access nZEDb install wizard.

your-server-ip/install

or

your-domain.com/install

Setting up Nginx Server Block

Create a new server block file in /etc/nginx/conf.d/ directory for nZEDb.

sudo nano /etc/nginx/conf.d/nzedb.conf

Put the following text into the file. Replace 12.34.56.78 with your Ubuntu server’s IP address. You can also use a domain name.

server {
    listen 80;
    server_name 12.34.56.78;

    access_log /var/log/nginx/nzedb-access.log;
    error_log /var/log/nginx/nzedb-error.log;

    root /var/www/nZEDb/www/;
    index index.html index.htm index.php;

    location ~* \.(?:css|eot|gif|gz|ico|inc|jpe?g|js|ogg|oga|ogv|mp4|m4a|mp3|png|svg|ttf|txt|woff|xml)$ {
        expires max;
        add_header Pragma public;
        add_header Cache-Control "public, must-revalidate, proxy-revalidate";
    }

    location / {
        try_files $uri $uri/ @rewrites;
    }

    location ^~ /covers/ {
        # This is where the nZEDb covers folder should be in.
        root /var/www/nZEDb/resources;
    }

    location @rewrites {
        rewrite ^/([^/\.]+)/([^/]+)/([^/]+)/? /index.php?page=$1&id=$2&subpage=$3 last;
        rewrite ^/([^/\.]+)/([^/]+)/?$ /index.php?page=$1&id=$2 last;
        rewrite ^/([^/\.]+)/?$ /index.php?page=$1 last;
    }

    location /admin {
    }

    location /install {
    }

    location ~ \.php$ {
        include /etc/nginx/fastcgi_params;
        fastcgi_param  SCRIPT_FILENAME    $request_filename;

        # Uncomment the following line and comment the .sock line if you want to use TCP.
        #fastcgi_pass 127.0.0.1:9000;
        fastcgi_pass unix:/run/php/php7.0-fpm.sock;

        # The next two lines should go in your fastcgi_params
        fastcgi_index index.php;
    }
}

Save and close the file. Next, edit fastcgi_params file.

sudo nano /etc/nginx/fastcgi_params

Find the following line.

fastcgi_param  REDIRECT_STATUS    200;

Comment out this line by putting # symbol at the beginning.

#fastcgi_param  REDIRECT_STATUS    200;

Save and close the file. Then test Nginx configuration.

sudo nginx -t

If the test is successful, reload Nginx for the changes to take effect.

sudo systemctl reload nginx

Now visit the following URL to access nZEDb install wizard.

your-server-ip/install

or

your-domain.com/install

Finishing the Installation in Web Browser

Click the button at the bottom.  In the first step, the install wizard checks your PHP extensions and settings.

We already installed all needed PHP extensions earlier. For best performance, we also need to configure some PHP settings. If you use Apache and the PHP7 module, then open the following file.

sudo nano /etc/php/7.0/apache2/php.ini

If you use Nginx and PHP-FPM, then open the following file.

sudo nano /etc/php/7.0/fpm/php.ini

Change the value of max_execution_time from 30 to 120.

max_execution_time = 120

Change the value of memory_limit from 128M to 1024M.

memory_limit = 1024M

Define a timezone like America/New_York. 

date.timezone = America/New_York

Save and close the file. If you use Apache and PHP7 module, then you need to reload Apache for the above changes to take effect.

sudo systemctl reload apache2

If you use Nginx and PHP-FPM, then you need to reload PHP-FPM service for the above changes to take effect.

sudo systemctl reload php7.0-fpm

Now reload nZEDb wizard and all status should be OK. We can go to the 2nd step to set up the database. Enter the database details like so.

Step 3 is to set up OpenSSL. we need to download a certificate bundle to verify your Usenet provider’s certificates.

sudo wget -P /etc/ssl/certs/ http://curl.haxx.se/ca/cacert.pem

Then make it readable by all users.

sudo chmod 744 /etc/ssl/certs/cacert.pem

Now in the nZEDb setup wizard, enter the CA bundle path (/etc/ssl/certs/cacert.pem) and certificate folder (/etc/ssl/certs/). Make sure Verify peer and Verify host are checked. Do NOT check Allow self-signed certificates.

In step 4 enter the server details of your Usenet provider such as giganews.

Next, set up an admin account for nZEDb.

The final step is set up file paths. You can accept default values.

Indexing Usenet Content

Once the setup is complete, you will be able to visit nZEDb admin page and follow the instructions there to activate some newsgroups, get latest headers of your chosen groups and update releases.

To get latest headers, run

cd /var/www/nZEDb/misc/update

php update_binaries.php

Then create releases and NZB files using the following command.

cd /var/www/nZEDb/misc/update/nix/multiprocessing

php releases.php

Importing PreDB into MariaDB Database

Import daily dumps from nZEDb since 2014.

cd /var/www/nZEDb/cli

php data/predb_import_daily_batch.php 0 local true

Fetching Movie and TV Covers From IMDB, TMDB

To display covers on your nZEDb site, you need to get your own API key from IMDB or TMDB. Once you get your API key, go to nZEDb admin panel > site settings > edit site > 3rd party API keys and enter the API keys there.

I hope this tutorial helped you set up nZEDb Usenet indexer on Ubuntu 16.04/17.04.