Page tree
Skip to end of metadata
Go to start of metadata

Summary:

This document explains how to access the Point Weather Forecast (derived from BoM's ADFD - Australian Digital Forecast Database) data in Senaps. In particular this document focuses on using the THREDDS Data Manager (TDM) API in Senaps to extract specific point locations and return the data as JSON time series.

Prerequisites:

  • Senaps authorisation and license agreement. You must have Senaps credentials which have access to the ADFD data within Senaps. You can purchase a subscription through our Ag Climate Data Shop
  • An Understanding of how BoM ADFD data is structured. Please see the BoM ADFD User Guide document for details.

Please refer to the shop Terms and Conditions for specific terms relating to accessing ADFD data. 

Introduction

Senaps provides an API called TDM (THREDDS Data Manager) which allows for extracting point (longitude, latitude) location forecasts for ADFD data, sometimes referred to as 'drilling' the dataset.

TDM is a REST API with JSON encoded responses. 

Senaps provides a drill operation as part of the TDM API to provide an easy to use end-point for drilling gridded datasets. 

ADFD in Senaps

Currently the ADFD data is being synchronised from the BoM twice daily at 00:00Z (10am AEST) and 12:00Z (10pm AEST).

The latest set of files will always be available in our THREDDS data server catalog at consistent URLs. 

The listing of latest files is available as a web page:
https://senaps.io/thredds/catalog/org_catalogs/csiro/BoM_ADFD/latest/catalog.html

and also as an XML catalog:

https://senaps.io/thredds/catalog/org_catalogs/csiro/BoM_ADFD/latest/catalog.xml

Drilling ADFD - Example Test

As an example, we can use the interactive API documentation to test the drill operation.

The TDM drill API operation is documented here: https://senaps.io/api-docs/#/.  To access the correct API select THREDDS Data Manager API from the drop down box, enter your credentials (username + password or apikey) and select 'explore', and then click 'operations' to see the list of API endpoints.

Expand the GET /drill operation, by clicking on the 'GET' button, which allows you to interactively drill the time series for a given location for a given variable. Optionally, temporal constraints can be applied.

I.e. if we want to drill the MaxT_SFC variable we need to:

Step 1: Find the dataset path of the latest MaxT ADFD forecast.

    1. Look in the ADFD latest catalog (https://senaps.io/thredds/catalog/org_catalogs/csiro/BoM_ADFD/latest/catalog.html)
    2. select the file/variable of interest
    3. the dataset path is the shown in the URL: in this case: /csiro/BoM_ADFD/latest/IDZ71002_AUS_MaxT_SFC.nc
    4. Note these dataset paths are persistent and should not change.


Full catalog details

This is the full list:

/csiro/BoM_ADFD/latest/IDZ71000_AUS_T_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71001_AUS_Td_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71002_AUS_MaxT_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71003_AUS_MinT_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71004_AUS_Precip_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71005_AUS_DailyPrecip_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71006_AUS_Wind_Mag_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71013_AUS_PoP_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71014_AUS_DailyPrecip25Pct_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71015_AUS_DailyPrecip50Pct_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71016_AUS_DailyPrecip75Pct_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71017_AUS_Sky_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71018_AUS_RH_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71021_AUS_WindGust_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71022_AUS_WindWaveHgt_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71023_AUS_Swell_Mag_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71024_AUS_Swell2_Mag_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71030_AUS_DailyPrecip10Pct_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71031_AUS_Precip10Pct_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71032_AUS_Precip25Pct_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71033_AUS_Precip50Pct_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71034_AUS_WxIcon_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71068_AUS_ApparentT_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71069_AUS_SigWaveHgt_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71071_AUS_WindMagKmh_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71072_AUS_WindGustKmh_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71089_AUS_Wind_Dir_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71090_AUS_DailyPoP_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71091_AUS_RestOfDayPoP_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71092_AUS_Swell_Dir_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71093_AUS_Swell2_Dir_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71094_AUS_WxThunderstorms_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71096_AUS_WxPrecipitationFrozen_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71097_AUS_WxPrecipitation_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71102_AUS_WxFog_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71107_AUS_WxFrost_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71109_AUS_MixHgt_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71127_AUS_DF_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71152_AUS_DailyWxIcon_SFC.nc
/csiro/BoM_ADFD/latest/IDZ71153_AUS_HazWind.nc


NOTE: if you want to programmatically discover these dataset paths then the best approach will be to download the THREDDS catalog.xml from
https://senaps.io/thredds/catalog/org_catalogs/csiro/BoM_ADFD/latest/catalog.xml

(A JSON API for discovery gridded datasets is included in our development roadmap)


Step 2: Once you have your dataset path of you can query the drill API 

Having previously expanded the GET /drill operation you can now enter the parameters to drill the time series for a given location for a given variable. 

Fill in the mandatory path, longitude, latitude and variable fields as shown below (temporal constraints can optionally be applied).

  • full dataset path from above is /csiro/BoM_ADFD/latest/IDZ71002_AUS_MaxT_SFC.nc
  • as the BoM ADFD data set has one variable per file the variable name can be derived from the file name - the bit between the "_AUS_" and the ".nc" i.e. in this case it is MaxT_SFC. The one exception is the HazWind file who's variable name is HWA_SFC. The variable names are described in detail in the BoM ADFD user guide (http://www.bom.gov.au/catalogue/adfdUserGuide.pdf).

e.g.


Step 3: Execute request

The final request with authentication and parameters will look similar to below - the values for the slices at the nearest grid cell are contained in the 'results' array. The values are extracted directly from the netCDF file and are not manipulated at all by the drill operation.

Click on the 'Try it out!' button to execute the request.

If the request is successful the status code will be 200 and the response will contain the JSON encoded response similar to

{
  "datasetpath": "csiro/BoM_ADFD/latest/IDZ71002_AUS_MaxT_SFC.nc",
  "variable": "MaxT_SFC",
  "results": [
    {
      "t": "2019-05-15T00:00:00.000Z",
      "v": {
        "v": 17.2
      }
    },
    {
      "t": "2019-05-16T00:00:00.000Z",
      "v": {
        "v": 17.800001
      }
    },
    {
      "t": "2019-05-17T00:00:00.000Z",
      "v": {
        "v": 18
      }
    },
    {
      "t": "2019-05-18T00:00:00.000Z",
      "v": {
        "v": 17
      }
    },
    {
      "t": "2019-05-19T00:00:00.000Z",
      "v": {
        "v": 15.7
      }
    },
    {
      "t": "2019-05-20T00:00:00.000Z",
      "v": {
        "v": 17.1
      }
    },
    {
      "t": "2019-05-21T00:00:00.000Z",
      "v": {
        "v": 17.2
      }
    }
  ]
}

Directly using the API endpoint:

(Most of this can be deduced from the API Docs example above.)

a. The URL for the request is https://senaps.io/tdm/drill 
b. The request method is GET
c. Every request needs authenticating. Which can either be done either with a URL parameter API key, or a HTTP header API key or a HTTP Basic auth username and password. eg '&apikey=.....'
d. Example parameters can be seen in the api-docs and the above example.

The equivalent call to above is:

https://senaps.io/tdm/drill?path=%2Fcsiro%2FBoM_ADFD%2Flatest%2FIDZ71002_AUS_MaxT_SFC.nc&longitude=147.3&latitude=-42.99&variable=MaxT_SFC

The 'Try it out!' functionality in the API Docs also provides examples of making the API calls using the common command line tool curl.


API fair usage limits

To maintain quality and low cost service the ADFD-P product includes a fair use policy for API requests.

A single subscription of the ADFD-P product entitles the subscriber to 1000 requests per day to the TDM /drill end-point described in this document.

Please contact the Ag Climate Data Shop to discus options for higher usage.


  • No labels