If you have any comments, questions or feedback about these options, please contact researchdatasupport@csiro.au.
These are all the places that you may think about storing your research data. They are not all suitable or recommended. Scroll over for the full table
= Yes, 
= No, 
= Maybe.
Storage location | Recommended for working Research data | Recommended for long term archiving | Size limits | Access limits | Data Discoverable? | Available computing facilities | Linked to HPC? | Data protection | Old versions available | Further information and notes | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Local hard drive |
| | Hardware dependent | Only available to users with access to the PC | Only via folder names and structures, and file names. | Desktop or laptop | | weak | Only if backups managed personally | Data could be susceptible to loss without regular backups. See Rule 9 in Ten Simple Rules for Digital Data Storage | |
| External local drive or storage media (USB, Hard drive, CD, etc) |
| | Hardware dependent | Only available to who can access the drive | Only via folder names and structures, and file names. | | weak | Only if backups managed personally | Data susceptible to loss. See Rule 9 in Ten Simple Rules for Digital Data Storage | ||
| Physical project server |
| | Hardware dependent | Only available to who can access the server | | weak /moderate | Only if backups managed personally | Would not be able to be connected to IM&T networks. See Rule 9 in Ten Simple Rules for Digital Data Storage | |||
CSIRO IMT Services | Home drives |
For single person projects | | May be subject to quotas | May be available to CSIRO individuals on owners request | Only via folder names and structures, and file names. Not discoverable to others in the organisation. | | strong |
| See Storing and Managing Data | |
| Project Shared drives | | | No | Internal access only | Only via folder names and structures, and file names. Not discoverable to researchers outside the access group. | Possible for Windows HPC | strong |
| See Storing and Managing Data | ||
| Bowen Research Cloud (managed project storage) | | | No | Internal access only | With appropriate software layer | Ruby, Pearcey and Bracewell. Only data in Canberra can be used directly. | | strong | (mostly) | Bowen Research Cloud | |
| IMT Enterprise Virtual servers (VM) | | | No | Depends on the service running on the VM | Depends on the service running on the VM | moderate / strong | On request Will likely need own backup protocols implemented | Used for running services. | |||
| SharePoint |
File size limitations | | Users can request 0.5G, 1G, 2G or 5G. Not suitable for files greater than 250MB. File version control. | Can be used to collaborate with external partners | Searchable depending on access | Would need to move data in and out of SharePoint | Not directly | strong | | SharePoint | |
| Confluence |
File size limitations |
| Attachment size limit 100MB. No size limit per space. File version control. | Can be used to collaborate with external partners | Searchable depending on access | Would need to move data in and out of Confluence | | strong | | Confluence | |
| Data Access Portal |
Not recommended to store working data but could be used to create metadata record that points to your working data store to make it discoverable | | Collection size limit 1TB. No limits per project or per user. | Data can be shared internally, publicly or with specified users (including external) | Via collection level metadata | Can be done | strong | | |||
| Scientific Computing data store | |
Longest-serving large data store in CSIRO, since 1991. | No size limits. Default limit of 150,000 files. See: quota limits | All registered SC users | Primarily via POSIX filesystem, with folder name and structures, and file names. SAMBA (Windows). Web access available. | Ruby (direct-access), Pearcey and Bracewell | | strong |
| ||
| ePublish | | | File size limits 100MB | Not accessible except to corresponding author, approvers. ePublish reporters, and publication officers | Only searchable by associated manuscript metadata and by publication officers | | | strong | | Should not be used for data. | |
| Scientific Computing scratch (/flush*) file systems |
Large capacity and high capability - but copy important data to a safer place as soon as practical. | | Large quotas on inodes and space | All registered SC users | Access via POSIX filesystem, with folder name and structures, and file names. | Ruby, Pearcey and Bracewell | | moderate Hardware protection only | | Areas are subject to flushing, which removes old files. | |
Partner Services | NCI Raijin | |
Only in some areas | Depends on data type and filesystem | Can be public or only available to NCI users | For some areas | raijin tenjin cloud | | moderate / strong Varying with area | NCI backups depend on data type. | http://nci.org.au/services/data-management-storage/ |
AARNET CloudStor |
|
Your files will be accessible for as long as your institution login is valid | Two services: FileSender and Storage. FileSender used to send large files (encrypted). Default storage allocation 1TB. Can sync to local disk. | Users access CloudStor using their institutional account, through the Australian Access Federation (AAF). | | moderate / strong The AARNET CloudStor service does not come with any security documentation or rating. However, the login process is conducted using the AAF and the data transfer is encrypted using Secure Socket Layer (SSL). | | AARNet CloudStor | |||
Other External Services | External Consumer (Personal) cloud storage e.g. Dropbox |
Considerations and risks of using public cloud services | | Depends on service and plan. May be a cost. For example Dropbox has 2GB for free. 1TB for $13AU per month | Depends on service | moderate / strong with caveats May not be to CSIRO requirements | Often | Public Cloud (IMT) | |||
| External Cloud infrastructure and platform services e.g. AWS |
Considerations and risks of using public cloud services |
Considerations and risks of using public cloud services | moderate / strong with caveats | Maybe | |||||||
| External disciplinary and other certified trusted repositories | | with recorded approval from Rank 4 delegate | Depends on repository | Usually public | Yes | NA | Not to CSIRO facilities. May have linkages to cloud facilities. | moderate / strong with caveats May not be to CSIRO requirements | Depends on repository | See The PLOS One list of recommended repositories for examples. |
Column Definitions
Size limits | May come from policy/quota restrictions and/or physical or allocation size restrictions. Limits can often be raised if you have a justified need and there may be associated costs. File/object number limits may also apply. |
Access limits | Firewall and permission/control considerations. |
Data Discoverable? | A rough hierarchy of data discoverability might be:
Data accessible though services will tend to be more discoverable. |
Linked to HPC? | Can the data be used in place in a High Performance Computing workflow? |
Data Protection | Are systems in place that guard from single points of failure? Access control is assumed, but is usually weakest for portable devices. |
Old versions available | Is old content preserved if it is replaced by new content, offering some protection from mistakes? Is a history of changes kept? |
Recommended for working Research data | Working research data can potentially be recreated from source data by repeating a workflow. |
Recommended for long term archiving | Data that would be prohibitively expensive or impossible to reproduce. Take into account
The only recommendation in CSIRO is the DAP for this category. |
3 Comments
Bell, Robert (IM&T, Clayton)
Under 'Scientific Computing data store' in the final column 'Further information and notes', we could add.
Information about the latest backup of the data store and other user systems can be found at
https://hpc.csiro.au/users/backupstatus/ - link text 'User Filesystem Backup Status'.
Thanks
Rob.
Hannan, Katie (IM&T, Waite Campus)
Add in, which storage option can be locked down, by locked down do you mean restricted use Cook, Sue (IM&T, Kensington WA)?
Trend, Alice (CorpAffairs, Kensington WA)
Looks good Cook, Sue (IM&T, Kensington WA) - I can understand it!