How can you transition from photo-type or exploratory research software to production code? Can you? I've been discussing this with my guest Duncan McGregor. In the first part of our discussion we touch on how you can effectively analyse large data sets that are distributed over different locations with tools like Dask.

Links:

• https://www.youtube.com/@PairingWithDuncan Duncan's YouTube channel
• https://www.oneeyedmen.com Duncan's website
  • https://github.com/dmcg and his GitHub
• https://www.dask.org Dask
  • https://github.com/aws-solutions-library-samples/distributed-compute-on-aws-with-cross-regional-dask cross regional programming with Dask
  • https://aws.amazon.com/blogs/big-data/build-efficient-cross-regional-i-o-intensive-workloads-with-dask-on-aws/
  • https://blog.dask.org/2022/07/19/dask-multi-cloud
• https://zarr.readthedocs.io/en/stable/ Zarr format
• https://en.wikipedia.org/wiki/NetCDF NetCDF
• https://www.ecmwf.int European Centre for Medium Range Weather Forecasts
  • https://github.com/ecmwf
  • https://www.eumetsat.int
• https://pangeo.io
• https://testbook.readthedocs.io/en/latest/ Unit testing with Jupyter
  • https://github.com/nteract/testbook

Conferences/Events:

• https://dhrse2025.er.kcl.ac.uk King's College London: summer school for digital humanities. 30 June - 4 July, London
• https://www.software.ac.uk/workshop/collaborations-workshop-2025-cw25 The annual Collaborations Workshop of the Software Sustainability Institute UK, 13-15 May, University of Sterling, Scotland
• https://www.helmholtz-hirse.de/events/2025_06_13-rsehpcatisc.html The RSE workshop (13 June) at the IS

Get in touch

Thank you for listening! Merci de votre écoute! Vielen Dank für´s Zuhören!

Contact Details/ Coordonnées / Kontakt:

• Email mailto:peter@code4thought.org
• UK RSE Slack (ukrse.slack.com): @code4thought or @piddie
• US RSE Slack (usrse.slack.com): @Peter Schmidt
• Mastodon: https://fosstodon.org/@code4thought or @code4thought@fosstodon.org
• Bluesky: https://bsky.app/profile/code4thought.bsky.social
• LinkedIn: https://www.linkedin.com/in/pweschmidt/ (personal Profile)
  • LinkedIn: https://www.linkedin.com/company/codeforthought/ (Code for Thought Profile)

This podcast is licensed under the Creative Commons Licence: https://creativecommons.org/licenses/by-sa/4.0/