Q1 What was the biggest technical challenges to be able to participate in S2S/TIGGE projects at the beginning? What could improve next time?

Challenges:

  • conversion to required GRIB2 format is a challenge, specially from NetCDF for example
  • fixed reforecast production
  • numerous S2S parameters 

Ideas for improvement next time:

  • conversion tools could be shared among the centres (using NetCDF)
    • even within one organisation links to other projects and sharing of tools are important (KMA, BoM, CMA..)
  • conversion tools from ECMWF should be shared
    • new tools always welcomed (e.g. to convert to required accumulated sums since the forecast start)

Q2 What are the biggest challenges in the current production phase to ensure long-term minimal effort operations? How could the data archive centres support you more in that role?

Challenges:

  • conversion scripts upgrade following model's upgrades
    • sometimes even preventing to continue contribution to archive
      • => official project support stressing its importance  is helpful to find resources inside organisation
  • any operational issue can impact products for archives (which has not highest priorities)
  • fixed forecast computation is big job after each model upgrade (CPU, storage..)
  • TIGGE high resolution is really needed?
    • who is using it?  (limited domain with high res?)
      • some stats could help to understand if it is used

Q3 How could we improve cooperation between data providers and archiving centres (how to communicate/automate tasks/implement checking tools/test/implement model upgrades etc)?

  • to share available used tools and ideas (encoding check; value limits checks; input file checks (number of fields..))

  • notify each time even everything went well (automatic email alerts, some status files)

Q4 What are good/bad features in the design of S2S/TIGGE databases (data format/structure/encoding/compression etc)?

  • more complex GRIB packing might cause problems (higher CPU needed for data processing)
    • smaller file sizes for acquisition still can be achieved using standard unix compression like gzip

Q5 What are good/bad features in the interfaces for getting S2S/TIGGE data (web portals/Web API/direct MARS access etc)?

Problematic features:

  • GRIB to NetCDF conversion
    • generally should work in ECMWF data portal but it is not visible 
      • => format: "netcfd" in WebAPI should help
        • this should be documented
    • CMA provides on-the-flight conversion (using GRIB-API)
  • pre-computed data (e.g. eps means) before download (like IRI for S2S)
    • not planned at the moment (e.g. in Copernicus CDS (Climate Data Store))
    • can be partially done by MARS compute (for experienced users with ECMWF full access account)
  • slow access and problems with downloading the data
    • mainly from TIGGE survey 
      • big archive with data gaps affecting efficient data retrievement
    • S2S data access should improved a lot thanks to staging most data on disc (form tapes)
      • additional "tuning" still might help
  • interpolation is different in CMA & ECMWF archives
    • can impact user results based on the same original data

Good features:

  • interpolation!

Q6 What would be the most welcomed technical update of S2S/TIGGE databases from user point of view (data formats/data access/new products like time-series etc)?

  • addition of pre-computed values like eps means, weekly means, climatology computed from reforecasts..


Q7 Do you know other data archives from similar projects which could be inspiring next time (information exchange/interoperability/data standards etc)?

  • CMIP - tens of thousands users
  • Copernicus CDS

Q8 Are you generally satisfied with the way the S2S/TIGGE databases have been created and supported until now? What would you like to change or improve the most?

  • form surveys etc the archives seem to be very useful for scientific public
    • but to keep it working in long term official support is needed - especially for TIGGE which has been agreed until end of 2019 only at the moment
      • => workshop should stress this need

User survey summary

User survey summary - preliminary findings summarising examples of comments from the user survey (only issues not acknowledgements)