Research Computing and Data Bootcamp

It can be hard to know where to start with Research Computing and Data (RCD). There are many entry points and the learning curve can be intimidating. But once you get off the ground, an exciting new world of scientific analysis will be open to you!

This research computing and data bootcamp consists of a series of tasks and tutorials to familiarize you with research computing and data at NC State and beyond. It is entirely optional and self-directed, but if you run into trouble along the way, feel free to reach out to the RFS for assistance and we’ll be happy to help.

Set of foundational stairs and bricks

Let's get started

Linux Command Line

Many computational research tools require one of the more intimidating hurdles for novice users: the command line. However, getting through this hurdle opens up a whole new world of navigation and automation for research. 

What is the command line?

The command line interface (CLI) is a text-based program where you can enter commands that interact with a computer's operating system. You can use a CLI to communicate with your own laptop or another system like NC State's High Performance Computing (HPC) cluster. CLIs exist on Windows and MacOS, but much of research computing relies on navigating a Linux CLI. For example, you will need to be comfortable with Linux CLI to work on NC State’s High Performance Computing Cluster. 

Learning the command line will allow you to:

  • Run programs (yours and other researchers')
  • Navigate the contents of both your local system and other systems remotely
  • Access, edit, move and otherwise manipulate files

How to get started

This Linux for Beginners tutorial by Ryan Chadwick gives a basic introduction to Linux and the command line. We’d recommend working through it if you are new to working from the terminal and want to become a little more comfortable with research and computing that happens from a CLI. 

Ryans Tutorials: Linux Tutorial

Need a Linux environment?

If you aren’t already using Linux on your personal or work computer, you can still access a Linux terminal, although some tools will operate slightly differently in a true Linux environment.

If you’d like a dedicated Linux environment, you can access NC State’s HPC cluster by following the directions found here, or you can create your own virtual machine with Linux already set up through NC State's Virtual Computing Lab (video demonstration).

Git & GitHub basics

One you’ve started coding, how do you share your code with your collaborators and keep track of the way it changes and grows?

The near-universal solution for developers and researchers facing this problem is git. Git is version control software that tracks changes made to files in a folder (or “repository”), supported by hosted services like GitHub and GitLab. Aside from being a helpful tool for organizing, sharing, and tracking code you write, working with open source computing tools made by other researchers often requires some basic use of git, and understanding what’s happening under the surface can make the whole process run a lot more smoothly. 

How to learn Git

Git has created their own user-focused documentation, in the form of a book with explanations and exercises to help you get up to speed. The first two chapters present a solid introduction to what you can do with git and version control.

Pro Git book (Scott Chacon and Ben Straub)

If you prefer a video tutorial, Git Tutorial for Beginners: Learn Git in 1 Hour from Programming with Mosh on YouTube will walk you through concepts and basic usage.

Git Tutorial for Beginners: Learn Git in 1 Hour

Once you’re done with either learning option, try making a new repository on GitHub (you can also do so using NC State's cloud-hosted Enterprise GitHub environment) cloning it onto your local machine, making changes to a file, committing your changes, and pushing back to your remote repository. If you can do that, you’re well on your way to accessing, building, and sharing version-controlled code.

Globus

How do you get your files from where they are to where they need to be?

While many of the tools you are comfortable with probably use a drag-and-drop system to move files to remote locations, research computing tools are not always so intuitive. At NC State, our tool of preference is Globus. Globus is, at its core, a file transfer service that helps you move files quickly and reliably from point A to point B, specifically between Globus Endpoints. By configuring your local machine as a Globus Endpoint through Globus Connect Personal, you can

  • quickly move files to any other Globus Endpoint (including NC State’s Research Storage, Google Drive, and the HPC Cluster Hazel)
  • set up automatic transfers that occur at regular intervals to keep your research data synced and backed up.

Get started with Globus

Globus guide to getting started with Globus Connect Personal

Make a guest collection

Globus also enables you to share data on Globus Endpoints with your collaborators through Globus Guest Collections. This takes a specific folder/directory on a Globus Endpoint and allows you to grant access to a list of Globus IDs (typically institutional email addresses) or Globus Groups.

Take a look at the documentation on Globus Guest Collections and Globus Groups to familiarize yourself with the tools available, then try making a Guest Collection yourself. You can also follow along with this brief video demonstration.

Set up a transfer

Once you have Globus connect personal set up on your local machine and your choice of Guest Collection or Globus Endpoint, you can set up one-time or schedule syncs/transfers between your machine endpoints, including your Globus Connect Personal installation on your local machine. You can see some example options in this Globus transfer video demonstration, and some specific documentation for Globus transfers from Google Drive in this OIT guide.

NC State's Research Storage

Our final step in getting your Research Computing and Data (RCD) bearings at NC State is getting set up on NC State’s Research Storage. While you're not required to use NC State's Research Storage, we find it to be a helpful resource to know about as you collect and generate data through your research.

We highly recommend using NC State's Research Storage if you have research data that needs to be securely stored or backed up.

What is Research Storage?

Research Storage is a service offered by the university that keeps your research data safe, secure, and backed up. It can be accessed both on campus and off (although you will need to use the VPN when off campus) and is also available across NC State’s HPC cluster Hazel. 

Who can use it?

Each individual faculty member is eligible for a baseline of 2 terabytes AND 2 terabytes per funded project, both of which can be expanded on request. 

While non-faculty researchers cannot directly request Research Storage, faculty members can grant full access to any other NC State users who have a unity ID through the OIT Research Services web portal, or they can set up more finely-grained access to specific folders through Globus Guest Collections.

For an overview of research storage and information on accessing it, see the Research Storage documentation.

Request a storage allocation (faculty only)

Log into the OIT Research Services web portal to set up your storage allocation. See OIT's Research Storage documentation for more details on:

OIT Research Services web portal

How to access your allocation

Once you have an allocation on Research Storage, try mounting it to your work computer or a virtual machine through NC State's Virtual Computing Laboratory. Once mounted, it should operate similarly to a typical networked or external drive. Note that if you are off campus, you will need to be logged on to the NC State VPN.

Mounting instructions for NC State's Research Storage

Questions?

If you have any questions or are in need of further clarification, feel free to contact the RFS or reach out to research storage directly at oit_research_storage@help.ncsu.edu.

Next steps

You did it! Or you skipped to this part of the bootcamp. We can’t tell. Either way, with the basics under your belt you are ready to get back to what you love doing: your research. 

Next steps to use what you know:

Librarians

We're glad! Thanks for the feedback.

Website feedback

What is the issue with this page?

We hear you! Thanks for letting us know.