The uow command

The usage of the uow program is divided into several subcommands.

The bootstrap subcommand

Before any uow directories can be created, the program needs to be configured with one piece of information:

Warning

The bootstrap process does not check to see if the api end point exists, or if your api key is valid. It only creates the config file.

The init subcommand

The init subcommand is what actually creates a uow directory. It has two options:

  • -e, --expocode is the expocode of the cruise you want to create a uow directory for.

  • -d, --dir_name is the name of the directory which will be created.

If any option is not specified, it will be prompted for.

The uow directory name will be exactly as specified, there is no more implicit naming.

When run, the init subcommand will create the requested uow directory and all the default subdirectories. It will also GET the the cruise object referred by the given expocode and a list of files attached to this cruise.

Note

Unlike previous versions of the “fetch/commit” programs, files cannot be specified at uow directory creation time. No files are downloaded as part of the init process.

The list subcommand

The list subcommand shows information on the files attached to the cruise the uow directory is for.

The flag --raw will output the raw filtered json array.

The argument -t` specifies how to filter the list for display. Allowed values are:

all

Lists all the files attached to the cruise

dataset

Lists only the files which have the role dataset

unprocessed

Lists only the files which have the role unprocessed. This is equivalent to the “queue files” of the older systems.

merged

Lists only files which have the role of merged.

other

Lists files which have a role other than dataset, unprocessed, or merged. This would be non public files for argo, archive files, or hidden files.

The id listed by the list subcommand is what should be given to the fetch subcommand to download a file.

The argument -q allows for any jq boolean expression to filter the files by. The expression must always be contained with single quote chars '. A jq boolean expression is anything that produces a true/false value in jq, see https://jqlang.github.io/jq/manual/ for more documentation. This can query the entire file metadata record and not just what is displayed in the result table.

Examples:

List all files submitted by “Robert Key”:

uow list -q '.submissions[] | .name=="Robert Key"'

List all files submitted by someone with a case insensitive match on “key”:

uow list -q '.submissions[] | .name | test("key";"i")'

List all files submitted after 2019-01-01:

uow list -q '.submissions[] | .date>="2019"'

List all files without the dataset role:

uow list -q '.role=="dataset"|not'

The info subcommand

The list subcommand shows the entire json object for the cruise or requested file id. The one optional positional argument specifies the file json object to display.

If the file id is omitted, the cruise object for the current uow context is printed to stdout. These can be very long so it is recommended that this output be piped to your pager of choice.

The id listed by the list subcommand is what should be given to the info subcommand to view a file json object.

The fetch subcommand

The fetch subcommand is used to actually download files to the 0.existing_files subdirectory in the uow directory.

It takes one or more file ids as positional arguments.

It has four optional arguments:

--external

By default, the fetch subcommand will not allow fetching files not already attached to the cruise the uow is for. To override this limitation, simply include the --external flag. Before doing so, consider if the file should be attached to the cruise and perhaps do that first.

Note

The fetch subcommand only knows which files were attached to the cruise at the time of the init command. It will consider all files attached to the cruise after the uow was created to be external files.

This known file list can be updated with uow refresh

--panic

Sometimes the data is so strange, you just need to grab all the files and sort through them manually. Of particular interest will be the archive.tar file, as this is the pre pycchdo data directory.

-q/--query

Download the all files matching the results of the query, see the uow list subcommand help for query usage

Run the query against uow list first to see what will be downloaded.

-t/--type

Download all files matching the filters from the type option of uow list.

Rule the type argument against uow list first to see what will be downloaded.

The status subcommand

The status subcommand shows several useful bits of information:

  • a list of files which have been fetched and if they are still present and unmodified

  • if a uow.json is present, it will be validated

  • if a valid uow.json is present, the anticipated results of a commit based on it will be listed

The status subcommand has no positional or optional arguments (other than help).

The commit subcommand

The commit subcommand actually performs the commit. It will do all the checks done by the status command. Additionally, it will query the API to see if files which should already exist do, and files which should not exist do not.

It has no positional or optional arguments (except for help).

After displaying the anticipated results of the commit, it will ask for confirmation to continue, any non “yes” response will abort the commit.

Danger

Do not abort the commit after saying “yes” to continue. Depending on how large the files to commit are, and what the bandwidth is, it may take a very lone time to complete.

Aborting a commit will cause any files which have already made it to the website to be orphaned.

Danger

If anything goes wrong, a dump of the last api call will be printed to stdout, include it with any bug reports.

Danger

Any interruption of the commit after saying “yes” to continue will require manual fixing.

The refresh subcommand

The refresh subcommand loads the cruise metadata and attached file metadata again.

This basically updates the files under the list subcommands and will allow fetching of files attached to the cruise after the uow was created without using the external flag.