Navigating Cloud Storage with Globus
September 01, 2022 | Susan Tussy
I was speaking with a colleague from another institution the other day. We were lamenting the lack of in-person events and actually missing traveling to visit colleagues at their home institutions. While this is certainly one of the major themes of the past two years, another theme that I’ve heard in the course of this new normal of virtual meetings (and the few in-person events, thank you Supercomputing) is mass migration of data between cloud storage platforms, and even between cloud stores and on-prem storage.
There are varying reasons for this mass migration, from security concerns to cost. Regardless of the reason, there are common barriers to success that can make these data transfers difficult.
The simple fact is that while they all have “cloud” in the name, either literally or philosophically, they are different storage systems. While the human interface to these systems is often browser based, the way they operate, and their data transfer workflows can be quite different. They are also not intrinsically designed to communicate with each other. This often necessitates a two-hop transfer, from cloud storage A, down to local storage, and then up to cloud storage B. With the file sizes we deal with in the big data science community, this can be a time-consuming proposition.
Globus has a connector for nearly every popular cloud storage out there. This allows for seamless abstraction of those storage types as a Globus Collection. The real beauty of this solution is that regardless of the storage type, they all look like one thing to the Globus Web App (and the CLI and API as well) and therefore present themselves to the user the same way. No need to fish around for a specific app to connect to that storage or remember how a particular workflow functions, all storage systems are Globus collections and Globus Transfer functions in the way Globus users are familiar with.
Globus has a connector for nearly every popular cloud storage out there. This allows for seamless abstraction of those storage types as a Globus Collection. The real beauty of this solution is that regardless of the storage type, they all look like one thing to the Globus Web App (and the CLI and API as well) and therefore present themselves to the user the same way. No need to fish around for a specific app to connect to that storage or remember how a particular workflow functions, all storage systems are Globus collections and Globus Transfer functions in the way Globus users are familiar with.
Tags
Related Content
Announcing The GlobusWorld Tour
We're excited to announce that we're taking GlobusWorld on the road! Over the next few months we will be conducting...
Hints On Data Storage And Transport From High Energy Physics
It may be surprising that with a PhD in biochemistry, Brigitte Raumann spends a lot of time these days considering data storage and transport in high energy physics and astronomy. But that also puts her in the perfect position to recognize some of the lessons from those disciplines that the life sciences can pick up. Bio-IT World sat down with Raumann to talk about the challenges she sees in data management in the life sciences and the solutions available.