Storage usage analysis tools

ncdu  tool to show map of usages from current directory down.

included in ubuntu / debian base

apt-get update
apt-get -y install ncdu

Openmediavault notes

Openmediavault is an opensource nas system that supports intel and arm.

Most notes here will be related to the Friendlyarm Nanopi Neo and Nanopi Neo2

First tests with v1.0 media dock had some problems.  The USB port on the dock board worked, but the original attempt to use a Seagate 2tb drive didn't work.

After much help from Friendlyarm and debugging the kernel, it appeared that there was a probably known bug manifested for the drive in this case.  Will be testing later with other drives and also new builds of the kernel as well to see if that problem works out.

Great support from Friendlyarm in working on this.

V1.0 JM20329 SATA -> USB bridge.  The board has a 12V supply which runs both the SATA drive and the NEO or NEO2 board.

Updates for V1.2 to follow.

The Friendlyarm NEO and NEO2 both have debian builds with openmediavault already built.  Most of the instructions for installing the OMV on the Intel or generic build up install can be ignored, because as supplied, the Friendlyarm images have been installed with generic settings, then the SD images saved.

The instructions below have instructions on a "from bare metal" install on Intel and perhaps others, and the info for configuring the Openmediavault post install from the web interface is the reason for including the links or information here.


Links:

Howto forge page on installing scratch openmediavault.

https://www.howtoforge.com/tutorial/install-open-media-vault-nas/

Current OMV version for NEO2 in use:

5/25/2017, link below

http://www.mediafire.com/file/89ugz0iig8jt6lv/nanopi-neo2_debian-nas-jessie_4.11.2_20170525.img.zip

Migrating Readynas Radiator to OS 6 Readynas devices

One challenge in migrating from Sparc architecture products, to Intel or other (Arm), is that the Linux file system on sparc was built with a 4k block size, rather than 512.

On Intel linux, if you set up your disks, you can mount them with some incantations and special procedures.  Not sure how the Readynas Linux handles that problem, but there is a link showing it.

I had a bad disk and some problems with the data recovery firmware on a REadynas NV+ some years ago and got the support group to help me get around it.  In the mean time, I had mounted the 4 drives on an Intel linux system I'd built up and had recovered most of the data.

https://kb.netgear.com/29875/ReadyNAS-Migrating-disks-from-RAIDiator-to-OS-6?cid=wmt_netgear_organic

Copy of article for archival:

This articles outlines the necessary steps to access data on disks from ReadyNAS units running RAIDiator on ReadyNAS OS 6 units.
Due to differences in CPU architecture and operating system between RAIDiator and ReadyNAS OS 6, it may be necessary to take additional steps once the disks are moved to an OS 6 chassis.
If your ReadyNAS running RAIDiator is still functional, and you can access the data, we recommended you first backup the data to another location
http://kb.netgear.com/app/answers/detail/a_id/21344
If you can no longer access the data using your legacy ReadyNAS and you wish to access the data using your ReadyNAS OS 6 device, booting may not be as easy. Depending on which legacy model the disks come from, and which ReadyNAS OS 6 you posses the steps differ to make the data accessible.
After initially moving the disks from the legacy ReadyNAS to the ReadyNAS OS 6 model please be aware of the following items;

• Do not attempt an OS reinstall
• Do not attemp a factory reset
• The RAID must be healthy i.e. if the RAID was broken on the legacy NAS it will not always be possible to access the data on the ReadyNAS OS 6 - an example of this would be 2 failed disks in a 4 disk RAID 5.
• ReadyNAS OS 6 model must have at least the same amount of drive bays as the legacy ReadyNAS
• Logging a ticket with Technical Support may be required
• Purchasing a data recovery contract may be required
• You may require one additional spare blank disk
• You may require an external location to where you can recover your data to i.e. external USB HD, network share, other NAS storage.

To find the steps needed to access the data on your ReadyNAS OS 6 device, find your model below:

• ReadyNAS OS 6
  • ARM
    • ReadyNAS 100 series (RN102, RN104)
    • ReadyNAS 200 series (RN202, RN204)
    • ReadyNAS 210 series (RN212, RN214)
    • ReadyNAS 2120 (RN2120)
  • x86
    • ReadyNAS 300 series (RN312, RN314, RN316)
    • ReadyNAS 500 series (RN516)
    • ReadyNAS 700 series (RN716X)
    • ReadyNAS 3130 (RN3130)
    • ReadyNAS 3138 (RN3138)
    • ReadyNAS 3220 (RN3220)
    • ReadyNAS 4220 (RN4220)

Find the model of your Legacy ReadyNAS running RAIDiator, click on the link and follow the instructions. X represents whether or not your model came with disks.

• RAIDiator
  • RAIDiator 4.1 (sparc)
    • ReadyNAS Duo (RND2XXX)
    • ReadyNAS NV+ (RND4XXX)
    • ReadyNAS 1100 (RNR4XXX)
    • Follow the instructions at ReadyNAS: Migrating disks from RAIDiator 4.1 or RAIDiator 5.3 to ReadyNAS OS 6 
    • http://kb.netgear.com/app/answers/detail/a_id/29876 
• RAIDiator 4.2 (x86)
• • ReadyNAS NVX, NVX Pioneer (RNDX4XXX, RNDX4XXE)
  • ReadyNAS Pro Series, Pro Pioneer, Pro Business (RNDP2000, RNDP4XXX , RNDP6XXX, RNDPXXXXE)
  • ReadyNAS Ultra Series (RNDU2XXX, RNDU4XXX , RNDU6XXX)
  • ReadyNAS Ultra Plus series (RNDP2XXU, RNDP4XXU, RNDP6XXU)
  • ReadyNAS 1500 (RNRX4000)
  • ReadyNAS 2100 (RNRX4000)
  • ReadyNAS 3100 (RNRP4000)
  • ReadyNAS 3200 (RN12PXXXX)
  • ReadyNAS 4200 (RN12TXXXX, RN12GXXXX, RN12SXXXX)
  • Follow the instructions at ReadyNAS: Migrating disks from RAIDiator 4.2 to ReadyNAS OS 6 (x86)
  • http://kb.netgear.com/app/answers/detail/a_id/29957
• RAIDiator 5.3 (ARM)
• • ReadyNAS Duo v2 (RND2XXXv2)
  • ReadyNAS NV+ v2 (RND4XXXv2)
  • Follow the instructions at ReadyNAS: Migrating disks from RAIDiator 4.1 or RAIDiator 5.3 to ReadyNAS OS 6
  • http://kb.netgear.com/app/answers/detail/a_id/29876 
  
  

NFS Client and Server setup pages

To set up NFS Client and Server functions on Debian, et. al. here are some notes

NFS Server

apt-get install nfs-kernel-server nfs-common

NFS Client

apt-get install nfs-common

Exports

/home/jws           192.168.0.101(rw,sync,no_subtree_check)
/var/www        192.168.0.101(rw,sync,fsid=0,crossmnt,no_subtree_check,no_root_squash)

Other setup details related to mount @ boot time, etc., refer to link below.

Examples above from:

https://www.howtoforge.com/install_nfs_server_and_client_on_debian_wheezy

Readynas Radiator 4.1 firmware problem with SMB CIFS shares

The changes made to SMB protocol from version 1 to version 3, seems to have killed the share mode of access on readynas older systems.

Switching to user share mode and creating a user is a pain as well.

But the share name is entered as a user id in passwd, but not put up as a user in the gui, which is a huge hole when you switch from share to user.

the only method of fixing it is to install EnableSSHAccess and log in as root.  One can then set an SMB passwd on the shares that exist, and each will be accessible with that passwd on system thru so far and including windows 10.

Also other secured user access is then possible once the sharing is turned on if desired.  The users cannot access others storage w/o using group access features.

Of course this has a possibility of voiding any service you have with Netgear (warranty for 4.x devices is long gone).  But if you have any service agreements, don't do this w/o checking since it may impact your agreements for unauthorized user mods to devices.

To install ssh access for 4.x radiator do the following

Download EnableRootSSHAccess for your architecture from here

EnableRootSSHAccess

Original NV+ and 1100 systems are Sparc.  Pro and Ultra systems on 4.x will be Intel.  I'm not sure if Arm has a 4.x version.  All the ones I've seen with Arm are 6.0

then:

1. Get the SSH bin file from the ReadyNAS web site. Click the Add-ons for RAIDiator 4.1.3+ link and scroll to the EnableRootSSH link. Download the bin file and store it on your local computer.


2. Log in as admin on your ReadyNAS using a browser


3. Update your ReadyNAS firmware (optional). This is under System > Update. Just click the Check for Updates button.


4. Click the Local tab and click Choose File to find the EnableRootSSH.bin file that you previously downloaded. Upload this and follow the prompts.

 A restart of the Readynas system is required, if you have other users on the device.  (since some windows systems may work).  I doubt if any samba stacks such as one would have with shares on MacOS would work, but they may be impacted.

Install instructions from here;

http://blog.epdoc.com/2009/11/ssh-on-readynas-nv.html

CIFS mount problem on Linux, Error 12, can't allocate memory error

There is a workaround for this which is in the link below.

What worked however is this:

reg add HKLM\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters /v Size /t REG_DWORD /d 3 /f
sc stop  LanmanServer
sc start LanmanServer

Will have to research and update link with whatever is being done, but this worked for mounting my Google driver server windows 7 system.

Windows Xp so far has not shown the problem.  Other options have been necessary too, this is just the latest.

Occured on a Cubox debian Wheezy system on 3/27/2017 for me.

mount-cifs-mount-error12-cannot-allocate-memory

Readynas blog, with lots of reviews and info

http://www.rnasguide.com/2011/03/26/migrating-your-disks-from-one-readynas-to-another-readynas-on-the-same-platform/

possible workaround to readynas windows 10 share problem. (relevant to developer win 10 installs)

Note on removing credential (If I understand it), as a workaround fix for broken Windows 10 fix. 

Windows 10 upgrades to the developer releases seem to be invalidating stored credentials with each update with some regularity.  if you have a drive share mount with stored credentials, or on another front though not with the readynas nas, stored logins for say remote desktop, you have to reinput them.

That was working up to the latest 9/14 or 9/16 set of updates.  now it seems to retain the credential and not let you easily reinput the login, or share.

And it was not accepting any password on other shares, even valid ones.

A fellow on the Readynas forum (where this became a topic) suggested a workaround worth documenting.

Thanks to Readynas forum user nayadn for this.

nayadn (Aspirant) posted a new reply in R6000 Series AC WiFi Routers on 2016-09-17 07:32 AM :

---

Re: Can no longer access ReadyShare drives on Windows 10

i googled again the error and i came up with a sideway solution
go into user account in control panel
if you're in windows 10 select manage your credentials this will open 2 options
web credentials and windows credentials
head to windows credentials and add a new windows credential
enter the location of the drive \\readyshare on the first line
your windows user name in the second line
your windows password in the third line
and you're done, you should see the drive under network and be able to access it as before
good luck

Banana Pi sata port multiplier

Banana Pi Sata port multiplier

http://www.htpcguides.com/build-debian-image-for-banana-pi-pro-sata-port-multiplier/

following from above blog

Buildup for raid nas server using banana pi and port expander.

Build the Custom Debian Image for Banana Pi Pro SATA Port Multiplier

You are going to build a minimal Banana Pi debian image with port multiplier support enabled in the kernel as a module so it can be enabled and disabled by adding a single line to a configuration file. This requires a Ubuntu 64 bit environment, if you are on Windows you can create a virtual machine using Virtualbox or VMPlayer to create the virtual machine.

SSH into your virtual machine or run the terminal directly on it.

Make sure you have git installed

sudo apt-get install git build-essential -y

Git clone Igor’s Banana Pi build scripts

git clone https://github.com/igorpecovnik/BananaPI-Debian

Edit the main build script

cd BananaPI-Debian
sudo nano build.sh

Change BRANCH to use the mainline latest and greatest updates denoted by next

BRANCH="next"

Adjust kerneltag so it uses 3.19.7, eventually it will be able to use Kernel 4.0

KERNELTAG="v3.19.7"

Now build the kernel

sudo bash build.sh

You will eventually be prompted for some configuration options, I just accepted the defaults by pressing Enter.

You will now have a working Banana Pi Pro image that supports WiFi and non-WiFi models and modular SATA port multiplier support!

The image that you can burn to your sd card sitting inside a zip file called Bananapi_Debian_3.0_wheezy_3.19.7.zip.

When it’s done transfer the image zip onto a USB and unpack it (hint you can connect USB devices to the Virtual machine). You will see a .raw file you can rename to .img and flash with Win32Imager, Apple Pi Baker or whatever you usually use to write the image to the SD card.

For those curious, you can see the configuration file which enables the SATA port multipler, we are using the next branch so it’s this file

sudo nano lib/config/linux-sunxi-next.config

CONFIG_AHCI_SUNXI= should be set to m for modular

CONFIG_AHCI_SUNXI=m

I also set these options which enable the PMP and are meant to help with input output speeds (SCHED)

CONFIG_SATA_PMP=y
CONFIG_SCHED_MC=y
CONFIG_SCHED_SMT=y

Ctrl+X, Y and Enter to save the configuration

After running build.sh the first time you already have the kernel and image. After you make changes like the ones above, you rebuild it using the same command, this time it will go much faster (like 15 minutes) to update the image with your new kernel configuration.

sudo bash build.sh

Test Banana Pi Pro SATA Port Multiplier

SSH in to the Banana Pi with your custom image, the default username is root with password 1234

Use this command to make sure the compilation was successful in including the module we needed

zgrep CONFIG_AHCI_SUNXI /proc/config.gz

It should output

CONFIG_AHCI_SUNXI=m

To enable the module for the port multiplier open this file

nano /etc/modprobe.d/ahci-sunxi.conf

Add this line

options ahci-sunxi enable_pmp=1

Ctrl+X, Y and Enter to save

Reboot with your hard drives connected and the jmn321 or other port multipler connected to power

sudo reboot

Use blkid to see the hard drives show up

blkid

Look at those drives ready for media content

Now I can see the 4 x 2 TB Samsung drives I bought for this project.

root@bananapi:~# blkid
/dev/mmcblk0p1: UUID="a73fb24c-de15-4a1f-a4fe-b6436f8d98f3" TYPE="ext4"
/dev/sda1: LABEL="HTPC Guides 1" UUID="190D-153C" TYPE="vfat"
/dev/sdb1: LABEL="HTPC Guides 2" UUID="1917-2940" TYPE="vfat"
/dev/sdc1: LABEL="HTPC Guides 3" UUID="19E3-1C09" TYPE="vfat"
/dev/sdd1: LABEL="HTPC Guides 4" UUID="1911-0E2F" TYPE="vfat"

If you ever want to disable the port multiplier, just remove the options ahci-sunxi enable line.

Enable WiFi on Banana Pi Pro with Igor’s Image

Backup the dtb module just in case and overwrite the dtb module with the Banana Pro one

cp /boot/dtb/sun7i-a20-bananapi.dtb /boot/dtb/sun7i-a20-bananapi.dtb.bak
mv /boot/dtb/sun7i-a20-bananapipro.dtb /boot/dtb/sun7i-a20-bananapi.dtb

Enable the WiFi module

modprobe brcmfmac

Generate a hashed passphrase, replace SSID with your wireless router’s SSID and wifipassword with the actual password to access WiFi.

wpa_passphrase SSID wifpassword

You will get output like this

network={
ssid="HTPCGuidesSSID"
#psk="htpcguides"
psk=c78152ab34bbac9fa172f0572a63d313195da3763d0eeef8ee903ee9c921f8c5
}

Copy the psk line without the psk= part

Now we need to add the configuration to your network interfaces file

nano /etc/network/interfaces

Uncomment your wlan0 section and add the SSID and psk hash you generated before

# Wireless adapter #1
auto wlan0
        allow-hotplug wlan0
        iface wlan0 inet dhcp
        wpa-ssid HTPCGuidesSSID
        wpa-psk c78512ab34b3ac9f2172f0572a63d313195da3763d0e4ef8ee903ee9c921f8c5

Ctrl+X, Y and Enter to save

Reboot

reboot

Use iwconfig to show that the wireless is connected

iwconfig

You will see the wireless overview for the Banana Pi

wlan0     IEEE 802.11bgn  ESSID:"HTPCGuides"
          Mode:Managed  Frequency:2.412 GHz  Access Point: 10:7B:FF:EA:47:D5
          Bit Rate=58.5 Mb/s   Tx-Power=1496 dBm
          Retry short limit:7   RTS thr:off   Fragment thr:off
          Encryption key:off
          Power Management:on
          Link Quality=62/70  Signal level=-48 dBm
          Rx invalid nwid:0  Rx invalid crypt:0  Rx invalid frag:0
          Tx excessive retries:0  Invalid misc:0   Missed beacon:0

Banana Pi Pro Port Multiplier Image

This image will work out of the box. It will resize your sd card upon the first boot and regenerate SSH keys. The default SSH login is root with password htpcguides that you will be forced to change on the first login. There is also an Media Server Installer which I will post how to use in a day or two (it is in /root/HTPCGuides, more info here)

cubox setup notes

with the debian jesse of 3/9/2015, I had to set up the network manually.

copy the contents of /etc/network/intefaces.d/eth0 to the new image.

apt-get -y install cifs-utils  (still work in progress)

apt-get -y install nfs-common nfs-server to get nfs stuff set up

cifs mounts are not running, but smbd is running fine.  Still to gen up further testing as of 4/12/15

Nice NFS setup page

https://help.ubuntu.com/community/SettingUpNFSHowTo

NFS server

To check the NFS server is not installed, you can do...

• $ dpkg -l | grep nfs-kernel-server

Install the required packages...

# apt-get install nfs-kernel-server

NFSv4 client

Install the required packages...

• # apt-get install nfs-common 

On the client we can mount the complete export tree with one command:

• # mount -t nfs -o proto=tcp,port=2049 nfs-server:/ /mnt

  Note that nfs-server:/export is not necessary in NFSv4, as it is in NFSv3. The root export :/ defaults to export with fsid=0.

We can also mount an exported subtree with:

• # mount -t nfs -o proto=tcp,port=2049 nfs-server:/users /home/users

To save us from retyping this after every reboot we add the following line to /etc/fstab:

nfs-server:/   /mnt   nfs    auto  0  0

It may be required to force rpcbind to start.

this is accomplished by:

update-rc.d enable rpcbind

/etc/init.d/rpdbind start

The latter starts it for the current boot incarnation.  The update-rc.d makes it be

started on all subsequent boots till it is explicitly disabled.

Removing a Readynas 1100 or NV+ disk tray with bad button

The early Netgear Readynas NV+ (sparc) and 1100 units (4 across) all had a design which must have looked good on paper.

However the buttons that are supposed to push in, and displace a latch upward, allowing the arm to extract the drive typically is frozen.  I don't know if they ever worked early on, but nearly all of the original ifarant units and the first netgears, which all seem to be about the same just don't work.

there is a small spot in one of the holes you can stick a small probe into and actuate the latch thru a small lever action.

This video of my unit, while removed shows the actuation.  I'll share the raw video, a Youtube video of it, and since I'm not the first one to do this, another illustration for reference.

Disk Tray Removal, Youtube

Raw video (google drive)

A longer version of opening the Duo (and probably the NV)

Here is one of the newer units with a reviewer extracting the drive.

Readynas Duo (at about 2:19, has extract)

An annoying old review of the Readynas 1100 (for reference what one looks like)
Almost feel bad calling it annoying since the reviewer is a Jesuit priest.

Readynas 1100 Review