Quick start: Rich data management & sharing

Notable differences from other file sharing services

Using command line utility to manage data

foldershare is a command line client to manage data on this site, see instructions for using it on your computer and clusters like Comet, Stampede2 & Maverick.

1. Setup for your computer

1. Download the foldershare client and give it execute privilege, make a note of the location where you keep this client

   curl -OL https://seedmelab.org/sites/default/files/downloads/foldershare_php_client.1.2.0.tar.gz 
   tar xzf foldershare_php_client.1.2.0.tar.gz
   cd foldershare_php_client.1.2.0
   chmod +x folder-share
   

2. Download the my_seedmelab_conf.txt file, edit it with your credentials, move it to your home directory

   curl -OL https://seedmelab.org/sites/default/files/downloads/my_seedmelab_conf.txt
   mv my_seedmelab_conf.txt $HOME
   chmod 600 my_seedmelab_conf.txt

3. PHP7.2+ is required to use the foldershare client
   1. Mac OS ships with PHP.
   2. Windows follow instructions at noted here.
   3. Linux: If PHP is not available, then either build from source or install it as root, see instructions below.
      1. Build from source.
         Download PHP7.3.20 then unzip and build as follows

         % ./configure --disable-all --disable-cgi \
                     --enable-json --enable-cli --enable-mbstring \
                     --with-curl --with-readline --with-pcre-regex \
                     --prefix=/path2install/location/
         % make
         % make install
         % export PATH="/path2install/location:$PATH" # Add php to environment path
         

      2. Install with root privileges

         apt-get install php73-cli php73-curl php73-mbstring php73-openssl

4. Add foldeshare and PHP7.2+ path to your bash profile ~/.bashrc

       export PATH="/path/to/php72:/path/to/foldershare:$PATH"
       alias fs="foldershare --host $seedmelab_host --username $seedmelab_username --password $seedmelab_password"

5. See usage in sections 4-6

2. Comet cluster setup

1. Copy and edit my_seedmelab_conf.txt file to your home dir with host, username and password

   cp /software/community/seedmelab/conf/my_seedmelab_conf.txt ~

2. Add following alias to your shell environment file ~/.bashrc

   source ~/my_seedmelab_conf.txt; 
   export MODULEPATH=$MODULEPATH:/software/community/seedmelab/modules
   module load foldershare/1.2.0
   alias fs="foldershare --host $seedmelab_host --username $seedmelab_username --password $seedmelab_password"

3. Reload your shell environment

   source ~/.bashrc

4. See usage in sections 4-6

3. Stampede2 & Lonestar cluster setup

1. Copy and edit my_seedmelab_conf.txt file to your home dir with host, username and password

   cp /work/00521/tg457718/stampede2/seedmelab/conf/my_seedmelab_conf.txt ~

2. Add following alias to your shell environment file ~/.bashrc

   source ~/my_seedmelab_conf.txt
   export PATH="/work/00521/tg457718/stampede2/seedmelab/bin:$PATH"
   alias fs="foldershare --host $seedmelab_host --username $seedmelab_username --password $seedmelab_password"

3. Reload your shell environment

   source ~/.bashrc

4. See usage in sections 4-6

4. FolderShare command set

You will need this temporary site's host name and your user name and password in order to connect to this site. For instance, you can list top level folders like this:

./foldershare --host hostname --username login --password password  ls /

Many Linux-style file and folder commands are supported, such as:

• "ls" to list a folder's contents.
• "stat" to get the status of a file or folder.
• "mkdir" to make a new folder.
• "rm" to remove a file or folder tree.
• "rmdir" to remove an empty folder.
• "cp" to copy a file or folder.
• "mv" to move a file or folder.

There are a few more commands particular to web services, such as:

• "download" to download a file or folder.
• "upload" to upload a file or folder.
• "update" to change a file or folder's description.

5. Using the foldershare client

Requirement: fs alias must be set in your environment as noted in sections above

fs --help

# See help for a command
fs mkdir --help

fs mkdir /123
fs mkdir /123/child

# Update description for remote path /123
fs update description 'hello world' /123

# Upload file or folder to remote path /123
# wildcards not supported
fs upload --sync /local/readme.txt /123 # --sync option uploads missing files from local to remote destinations

# Download a folder (use absolute paths)
fs download /123 # downloads folder to current dir
fs download --sync /123 /tmp # --sync option downloads missing files from remote to local destinations

# Get description and field information
fs --format text stat /123
fs --format keyvalue stat /123

6. Using the foldershare interactive shell

Login to the site, this will drop the user in a foldershare shell, which enables easier manual interaction

foldershare --host hostname --username login --password password 

Create a new folder say "myfolder" at the top level (Will be available under Personal files when visited via the web browser).

foldershare> mkdir --help
foldershare> mkdir /myfolder

Upload a local file to the site using the upload command 

foldershare> upload --help
foldershare> upload /full/localpath/omega.zip /myfolder

Add a description to a file using the update command

foldershare> update --help
foldershare> update description 'this is metadata' /myfolder/omega.zip

Download a folder using the download command, this will zip and download the contents if appropriate

foldershare> download --help
foldershare> download /myfolder /full/localpath