r2r_ctd.reporting

Classes and functions that iterate through stations doing QA tests, aggregating the results, and reporting those results in the form of and XML element tree.

Because we are making XML, the code in here is a little verbose. The ResultAggregator is what iterates though stations, performs or asks for the QA results for each station, and constructs the final QA “certificate”. The builder pattern from lxml is being used here to allow the code to look similar to the XML that it is generating. If you are looking at the code yourself, start with ResultAggregator.certificate() and follow it from there.

Attributes

E	lxml element maker with the r2r namespace configured, a whole bunch of Element instances follow representing different XML elements that will be constructed
Certificate
Rating
Tests
Test
TestResult
Bounds
Bound
Infos
Info
Update
Process
Time
Reference
ALL	Literal representing 100 percent
A_FEW	Literal representing 50 percent, defines the cutoff between "yellow" and "red" ratings in individual tests
RATING_CSS_MAP	Mapping between the QA letter codes and css color name

Classes

ResultAggregator

Dataclass which iterates though all the stations their tests and aggregates their results and generates the "info blocks".

Functions

overall_rating(→ lxml.etree._Element)	Given a string code rating, wrap it in a Rating with the correct description attribute set
file_presence(→ lxml.etree._Element)	Constructs the XML element representing the "Presence of All Raw Files" test result
valid_checksum(→ lxml.etree._Element)	Constructs the XML element representing the "Valid Checksum for All Files in Manifest" test result
lon_lat_range(→ lxml.etree._Element)	Constructs the XML element representing the "Presence of All Raw Files" test result
date_range(→ lxml.etree._Element)	Constructs the XML element representing the "Dates within NAV Ranges" test result
boolean_span_formatter(→ str)	Format a boolean with html span element that colors green/red for true/false
get_update_record(→ lxml.etree._Element)
get_new_references(→ list[lxml.etree._Element])	Return a list of new Reference xml elements
write_xml_qa_report(breakout, certificate)

Module Contents

r2r_ctd.reporting.E

lxml element maker with the r2r namespace configured, a whole bunch of Element instances follow representing different XML elements that will be constructed

r2r_ctd.reporting.Certificate

r2r_ctd.reporting.Rating

r2r_ctd.reporting.Tests

r2r_ctd.reporting.Test

r2r_ctd.reporting.TestResult

r2r_ctd.reporting.Bounds

r2r_ctd.reporting.Bound

r2r_ctd.reporting.Infos

r2r_ctd.reporting.Info

r2r_ctd.reporting.Update

r2r_ctd.reporting.Process

r2r_ctd.reporting.Time

r2r_ctd.reporting.Reference

r2r_ctd.reporting.ALL = 1

Literal representing 100 percent

r2r_ctd.reporting.A_FEW = 0.5

Literal representing 50 percent, defines the cutoff between “yellow” and “red” ratings in individual tests

r2r_ctd.reporting.overall_rating(rating: Literal['G', 'R', 'Y', 'N', 'X']) → lxml.etree._Element

Given a string code rating, wrap it in a Rating with the correct description attribute set

r2r_ctd.reporting.file_presence(rating: Literal['G', 'R'], test_result: float) → lxml.etree._Element

Constructs the XML element representing the “Presence of All Raw Files” test result

Parameters:

test_result – Should be a string or int in the interval (0, 100) representing the percentage of files that passed this test.

r2r_ctd.reporting.valid_checksum(rating: Literal['G', 'R']) → lxml.etree._Element

Constructs the XML element representing the “Valid Checksum for All Files in Manifest” test result

Note that this check is pass/fail

r2r_ctd.reporting.lon_lat_range(rating: Literal['G', 'R', 'Y', 'N', 'X'], test_result: float) → lxml.etree._Element

Constructs the XML element representing the “Presence of All Raw Files” test result

Parameters:

test_result – Should be a string or int in the interval (0, 100) representing the percentage of files that passed this test.

r2r_ctd.reporting.date_range(rating: Literal['G', 'R', 'Y', 'N', 'X'], test_result: float) → lxml.etree._Element

Constructs the XML element representing the “Dates within NAV Ranges” test result

Parameters:

test_result – Should be a string or int in the interval (0, 100) representing the percentage of files that passed this test.

r2r_ctd.reporting.boolean_span_formatter(tf: bool) → str

Format a boolean with html span element that colors green/red for true/false

r2r_ctd.reporting.RATING_CSS_MAP

Mapping between the QA letter codes and css color name

class r2r_ctd.reporting.ResultAggregator

Dataclass which iterates though all the stations their tests and aggregates their results and generates the “info blocks”.

It is structured in the same order that the results appear in the XML. Some ratings require extra information, e.g. the geographic bounds test needs to know if any of the stations are missing nav entirely or if the bounding box itself is missing.

breakout: r2r_ctd.breakout.Breakout

geo_breakout_feature()

If the breakout has a valid bounding box, generate the GeoJSON feature to plot on a map

geo_station_feature()

Generate the GeoJSON feature collection with a feature for each station that has lon/lat coordinates to plot on a map

property presence_of_all_files: float

Iterate though the stations and count how many have all_three_files()

property presence_of_all_files_rating: Literal['G', 'R']

Pass/fail result string of presence_of_all_files() where 100 is a pass

property valid_checksum_rating: Literal['G', 'R']

Pass/fail result string of manifest_ok()

property lon_lat_nav_valid: float

Iterate though the stations and count how many are lon_lat_valid()

property lon_lat_nav_range: float

Iterate though the stations and count how many are lon_lat_in() the bbox()

property lon_lat_nav_ranges_rating: Literal['G', 'Y', 'R', 'N', 'X']

Calculate the rating string for the nav bounds test, also needs to check if all of the stations are missing nav or if the breakout is missing bounds.

property time_valid: float

Iterate though the stations and count how many are time_valid()

property time_range: float

Iterate though the stations and count how many are time_in() the temporal_bounds()

property time_rating: Literal['G', 'Y', 'R', 'N', 'X']

Calculate the rating string for the temporal bounds test, also needs to check if all of the stations are missing time or if the breakout is missing bounds.

property rating

Determines the overall color rating.

Color Codes:

• R (Red)

• N (Grey)

• X (Black)

• Y (Yellow)

• G (Green)

If any of the test results are GREY or BLACK, those ratings override the score based rating.

GREEN (G) if 100% of tests PASS, YELLOW (Y) if more than 75% of individual tests PASS, RED (R) if 75% or fewer of individual tests PASS; GREY (N) if no navigation was included in the distribution; BLACK (X) if one or more tests could not be run.

property info_total_raw_files

Info Element with the length of r2r_ctd.breakout.Breakout.hex_paths

property info_number_bottles

Info Element with the number of casts that have bottles fired

WHOI Divergence

The original WHOI code simply checks if there is a .bl file and says the cast has bottles fired if this file is present.

This is incorrect, you need to check to see if there are any actual bottle fire records in that file. This code does that check.

In the example breakout 138036 there are three casts but only one fired bottles, the QA report in that breakout incorrectly says 0 casts with bottles fired. My understanding is the current WHOI code would report 3 for this breakout, I don’t know why is says 0 but both are incorrect.

property info_model_number

Info Element with the CTD model number (e.g. SBE911)

See r2r_ctd.derived.get_model()

property info_number_casts_with_nav_all_scans

Info Element with the number of casts that have the string “Store Lat/Lon Data = Append to Every Scan” in the header file

property info_casts_without_all_raw

Info Element with a space separated list of station names that did not have all_three_files()

property info_casts_with_hex_bad_format

Always reports OK

WHOI Divergence

In the original WHOI code, this works the same way as info_casts_with_xmlcon_bad_format().

I would like to implement this in the same way that the bad xmlcon report does, but need to actually make or find some bad data.

property info_casts_with_xmlcon_bad_format

Report the casts which could not have a conreport generated

WHOI Divergence

The original code would run the file command and check to make sure any of “data”, “object”, or “executable” were not in the output of the command. Instead, this will just check to see which casts the ConReport.exe failed on, i.e. let seabird software figure out if the xmlcon is a bad format.

The original documentation also says it is looking for ASCII, but the code does not appear to do any encoding checks, they would likely be invalid anyway since Seabird files, being on windows, are usually CP437.

property info_casts_with_dock_deck_test_in_file_name

Info Element with a space separated list of station names that look like “deck tests”

See r2r_ctd.checks.is_deck_test()

property info_casts_with_temp_sensor_sn_problems

List of casts where the serial number in the data header does not match the first serial number for a temperature sensor in the xmlcon.

The seabird software throws an error if the above is not the case, this will prevent the creation of a cnv product even if the serial number is in the secondary channel.

property info_casts_with_cond_sensor_sn_problems

List of casts where the serial number in the data header does not match the first Conductivity sensor serial numbers in the xmlcon.

The seabird software throws an error if the above is not the case, this will prevent the creation of a cnv product even if the serial number is in the secondary channel.

property info_casts_with_bad_nav

Info Element with a space separated list of station names that aren’t lon_lat_valid()

property info_casts_failed_nav_bounds

Info Element with a space separated list of station names that are lon_lat_valid() but aren’t in bbox()

gen_geoCSV()

Generates the “geoCSV” file

The header was taken verbatim from the WHOI Code, and could probably use some cleanup. Of particular note is that the field types, units, etc.. metadata in the header, does not match the number of columns in the actual “data” part.

The original WHOI code also doesn’t calculate the dp_flag and just sets to a hard coded 0. Better might be to use a bit mask because there can be multiple problems with each cast.

property certificate

The Certificate Element with all the above test results

r2r_ctd.reporting.get_update_record() → lxml.etree._Element

r2r_ctd.reporting.get_new_references(breakout: r2r_ctd.breakout.Breakout) → list[lxml.etree._Element]

Return a list of new Reference xml elements

This crawls the output directories to check was was actually created to build its list

r2r_ctd.reporting.write_xml_qa_report(breakout: r2r_ctd.breakout.Breakout, certificate: lxml.etree._Element)