recursive extract from an Apache presented file structure via wget

when one finds a directory on the web which has such as

<parent dir>
directory
file1
file2

etc., it's desirable to extract the entire structure, recursing into the directory structure, duplicating the structure in some fashion locally

This example shows how to produce a local structure


https://stackoverflow.com/questions/23446635/how-to-download-http-directory-with-all-files-and-sub-directories-as-they-appear

wget -r -np -nH --cut-dirs=3 -R index.html http://hostname/aaa/bbb/ccc/ddd/

Explanation:

• It will download all files and subfolders in ddd directory
• -r : recursively
• -np : not going to upper directories, like ccc/…
• -nH : not saving files to hostname folder
• --cut-dirs=3 : but saving it to ddd by omitting first 3 folders aaa, bbb, ccc
• -R index.html : excluding index.html files

Reference: http://bmwieczorek.wordpress.com/2008/10/01/wget-recursively-download-all-files-from-certain-directory-listed-by-apache/

wget – recursively download all files from certain directory listed by apache

Case: recursively download all the files that are in the ‘ddd’  folder for the url ‘http://hostname/aaa/bbb/ccc/ddd/&#8217;
Solution:
wget -r -np -nH –cut-dirs=3 -R index.html http://hostname/aaa/bbb/ccc/ddd/
Explanation:
It will download all files and subfolders in ddd directory:
recursively (-r),
not going to upper directories, like ccc/… (-np),
not saving files to hostname folder (-nH),
but to ddd by omitting first 3 folders aaa, bbb, ccc (–cut-dirs=3),
excluding index.html files (-R index.html)

******************************
windows visual version
https://sites.google.com/site/visualwget/a-download-manager-gui-based-on-wget-for-windows


--30--

Install Lamp Server on Linux

Install Lamp server

sudo apt install apache2
sudo apt install mysql-server
sudo apt install php-pear php-fpm php-dev php-zip php-curl php-xmlrpc php-gd php-mysql php-mbstring php-xml libapache2-mod-php
sudo service apache2 restart

check php
php -r 'echo "\n\nYour PHP installation is working fine.\n\n\n";'



--30--

Installing lamp on a linux server (of course, the L is for that).

There is what seems to be a good page on this subject here

https://linoxide.com/linux-how-to/install-lamp-stack-ubuntu/

Transfering SQL database between two linux servers

Process for moving a database between two linux servers running mysql.

Target server may or may not have the database created.  example used uses phpmydmin create database to create the empty database.

Use mysqldump to create an ".sql" file with the data from the database.  Example here is for an RSS reader application, and keeps track of the rss feeds for doing deltas for the read presentation.

Note for sql the password and user id are for the mysql users and the password on the mysql server.  They are not linux account passwords.  The commands can be performed from any userid with access to the mysql commands, since the database is worked via the mysqldump command to the mysql socket.  Access to the actual files is not required, since the work is done on those files by the mysql server.

backup:

 mysqldump --password=<some-password> -u <userid> rss > 2016-0522-rss-mysqldump.sql

create database in phpmyadmin: 

Databases->Create new Database [enter rss in name field] (Create)

restore:

mysql -u <userid> --password=<password> rss < 2016-0522-rss-mysqldump.sql

Image index notes

Image index generator(s).

Imageindex:
http://www.edwinh.org/imageindex/

Image index web extensions
<needed>

installing lamp on ubuntu 12.04

one version of the install including installing and building up from scratch is here

http://www.ubuntugeek.com/step-by-step-ubuntu-12-04-precise-lamp-server-setup.html

this uses the text install  of 12.04 LTS to install Lamp and open ssh configured.

install webmin using the instructions here

http://www.ubuntugeek.com/how-to-install-webmin-on-ubuntu-12-04-precise-server.html

instructions for setting up on a 12.04 system w/o apache, php or mysql, putting up those parts after the fact is explained on this set of pages

https://library.linode.com/lamp-guides/ubuntu-12.04-precise-pangolin#sph_install-mysql

 installing various goodies, such as an extension to webmin to configure security is on the calpop site

http://calpop.com/blog.php

install splunk and ConfigServerFirewall

ConfigServerFirewall:

# wget http://configserver.com/free/csf.tgz
# tar xzvf csf.tgz
# cd csf

# ./install.sh

goto the Webmin Modules page and install a new module from a local directory. The module will be located at /etc/csf/csfwebmin.tgz.
Splunk

dpkg install splunk*.deb

Next, start Splunk: 

/opt/splunk/bin/splunk start

Now, enter http://your-server:8000 (where your-server is obviously the hostname or IP address of your box), and you'll enter the web interface.   You might well poke around for a bit.  After you've had your fill, and converted the license to the Free Edition (see the Splunk installation documentation for instructions on how to do this, by default, it runs as a 30 day trial of the paid version), log out again.

Whether you're managing one device or 1,000, Splunk (http://www.splunk.com/) is a useful product as it allows you to aggregate and search diagnostic information from a variety of systems.  At CalPOP we use it as a central syslog server, allowing us to view the logs of our several hundred Cisco and Juniper switches and other infrastructure elements in one central place, search for specific events, and build reports and dashboards to track performance.   If you're operating on a small scale, you can use the Free Edition of Splunk, which allows you to index up to 500 MB of data per day.   The Free Edition will likely cover you until your environment reaches enterprise-scale (think hundreds or thousands of servers), at which time Splunk will be more than happy to take your money.

The Free Edition of Splunk has one irritating drawback, however: it lacks any form of built in user account management or authentication.  We will (partially) address that shortcoming in the course of this tutorial.

First, download Splunk from their website, and upload it to the home directory your server.  If you're running Ubuntu, which is what we use for our infrastructure within CalPOP, you can use a .deb package.  There is also an .rpm for distros like CentOS and Fedora, and a tarball for everyone else.   Once you've uploaded it, if you're on Ubuntu (and presumably Debian, although I've yet to try it on that much-venerated distribution), run this command: