Developer’s guide

About

The Data Retriever Dashboard is divided into two parts:

  1. The Helper Script
  2. The Dashboard

The Helper Script

The script(dashboard_script.py in retrieverdash/retrieverdash/dashboard_script/ directory) is run regularly based on the cron job specification. It generates a dataset_details.json file containing the details of the datasets and the diff files(html files showing side by side diffs in the tables of a dataset if there is any change).

The Dashboard

The dashboard provides an interface where maintainers and users can see the details of datasets(whether it is installing successfully or not using retriever and the diffs).

Setting up locally

Before starting to make contributions to Data Retriever Dashboard you first have to set it up locally on your machine. You’ll need Python 3.4+. You have to run the script first so that you get the details of datasets generated to show on the dashboard.

Steps to run Data Retriever Dashboard locally:

  1. Clone the repository.

  2. From the directory containing manage.py, run the following command: pip install -r requirements.txt to install the requirements for the dashboard.

    To set up the dashboard and attach a cron job:

  3. python manage.py makemigrations to make migrations.

  4. python manage.py migrate to migrate.

  5. python manage.py crontab add to add the cron job for running the script that would check the installation of datasets.

    To setup the dashboard and start running the job regardless of the set cron configuration:

  6. If this is the first time you are running retriever on your computer, run retriever ls once the environment is setup.

  7. For developers on a local machine, run export RETR_TEST=true to switch to local testing mode.

  8. To run the cron for script immediately write the command python manage.py crontab show. Copy the hash for cron job. Then write python manage.py crontab run hash_of_the_cron_job. The script will start running immediately.

  9. Wait for the script to complete checking all datasets or open another terminal and go to directory where manage.py is and write python manage.py runserver to start the server for the dashboard. The default url is 127.0.0.1:8000. For using another port write python manage.py runserver port_number.

  10. Open a browser and load the url that was provided when you run python manage.py runserver . This is the dashboard.

Style Guide for Python Code

Run pep8 on the given file to make sure the file follows the right style. In some cases we do tend to work outside the pep8 requirements. The compromise on pep8 may be a result of enforcing better code readability. In some cases pep shows errors for long lines, but that can be ignored.

pep8 pythonfile.py

Testing

Follow these instructions to run a complete set of tests for any branch Clone the branch you want to test. Go where setup.py is located.

Install in development mode.

$  python setup.py develop

Running tests locally

To run tests we use pytest. From the retrieverdash directory where manage.py is. Run:

$   pytest -v

This will discover the tests and run it.

Continuous Integration

The main GitHub repository runs test on the Travis (Linux) continuous integration platform.

Pull requests submitted to the repository will automatically be tested using these systems and results reported in the checks section of the pull request page.

Documentation

We are using Sphinx. for the documentation. Sphinx uses reStructuredText as its markup language.

Update Documentation

The documentation is automatically updated for changes with in the dashboard_script.py and status_dashboard_tools.py. However, the documentation should be updated after addition of new modules. Make sure you check the changes and edit if necessary to ensure that only what is required is updated. Commit and push the new changes.

Test Documentation locally

cd  docs  # go the docs directory
make html # Run

Note:
Do not commit the build directory after making html.

Collaborative Workflows with GitHub

Submitting issues

Categorize the issues based on labels. For example (Bug, Important, Feature Request and etc..) Explain the issue explicitly with all details, giving examples and logs where applicable.

Commits

From your local branch of retrieverdash, commit to your origin. Once tests have passed you can then make a pull request to the retriever master (upstream) For each commit, add the issue number at the end of the description with the tag fixes #[issue_number].

Example:

Add more details to the dashboard

Skip a line and add more explanation if needed
fixes #3

Clean history

We try to make one commit for each issue. As you work on an issue, try adding all the commits into one general commit rather than several commits.

Use git commit --amend to add new changes to a branch.

Use -f flag to force pushing changes to the branch. git push -f origin [branch_name]

Working on the server

Data Retriever has a server that can be accessed with permission from the data retriever team. In order, to view the current dashboard that is running on the weecology server you will have to use the concept of port-forwarding. ssh -L 8000:localhost:8000 <user>@<server_domain>

tmux

On the server, you might also want to use tmux to keep the server running in the background. Usually, the tmux server is running with an instance name 0, just use:

retrieverdash@<servername>:~$ tmux ls
<instance name>: 1 windows (created Tue Aug  3 12:54:48 2021) [158x40]
retrieverdash@<servername>:~$ tmux attach -t <instance name>