Virtual Labs for Deep Learning with Audio Course
Learn more about using virtual dedicated lab environments for teaching a course.
Azure Lab services is a convenient service on Azure to quickly provision virtual machines for classrooms. We presented a use case of lab services for teaching a class in a previous blog post. In this post, we share another Aalto university course using the Azure lab services for student projects and exercises. Read on to learn about the scenario, challenges and the results.
We are providing Azure GPU virtual machine (VM) for a research project working with applying deep learning models to sound and music. We received satisfactory feedback from the research group regarding the use of Azure GPU-enabled (Nvidia V100) VM in their deep learning training. We decided to extend the same setup for a course in which students use different deep learning models applied to music and sound.
In this course, each student needs 24/7 access to a virtual machine to run, modify, access, control, input, output these deep learning models.
Different requirements were identified:
Each VM needs to have some minimum software/hardware requirements
Operating system: Linux | GPU: at least one GPU | RAM: at least 16 GB | CPU: 6-8 cores | Disk space: 250-500 GB
Have admin rights and ability to create Conda environment
About 15-18 students would need access to VMs for the duration of about 3 weeks
There was a need to transfer big audio files (few GB) between local machine and the VM
Cost effective and easy to manage environment
Deep learning training takes an arbitrary amount of time to finish (from minutes to days)
We set up Azure lab services and created a lab for this course. Azure Lab services provides a fast and flexible way to create, manage and remove virtual machines. It is important to learn about lab services, and its limitations before planning to use it.
For preparation, there are some questions to ask the course instructor before setting up the lab:
In Windows images, is Docker needed? (Helps to decide if we need nested virtualization or not)
Does the course instructor need to connect to student VMs later to evaluate their work? Or will students report in a different way? (Helps when configuring the VM login credentials)
What size of the VM and what operating system (OS) do you need? (Required to know when creating the lab and cannot be modified later)
Do you need GUI access to the VM? (Helps when configuring access to the VMs)
Even though we prepared in advance and had some prior experiences with Azure Lab Services, we faced two major challenges after setting up the lab.
Limited GPU sizes
The Lab services offer a limited group of VM sizes for labs. As mentioned in the background section above, the group was testing an Azure VM with Nvidia V100 GPU and were satisfied with the results. However, the V100 GPU was not available/supported in the lab. At the time we were creating the lab, the only GPU option for deep learning was Nvidia K80 GPU.
K80 is significantly slower than V100 and we realized the training takes much longer to finish. Fortunately, the course instructor was able to compromise on exercises and projects and adjust them accordingly.
We have already sent feedback to Microsoft about this challenge and the possibility of adding better VM sizes.
Update: Checking the VM sizes in mid-May 2022, the small GPU (for compute) size comes with Nvidia V100 now.
Tip: Make sure to leave sufficient time for testing the lab environment before starting the course.
We started testing the lab about 10 days before the beginning of the course and it left us some buffer to plan for changes.
Limited Disk space
The VMs also come with limited disk space. Lab Services supports image and disk sizes up to 128 GB. We needed at least 250 GB of space for each student VM. Unfortunately, this issue has been voiced in various places by different people, but there is still no concrete solution.
As a workaround, we used Azure Files share to provide extra storage for each student. This workaround adds more complexity to the environment for the admins and course instructors.
In overall, the course was successful, and students were able to practice all the materials without any issues. Students were onboarded with their email addresses and signed up to use their dedicated virtual machine from the Azure Lab Services portal.
11 registered students
Hourly cost of each VM: $ 1.39/hour
Total hours used (by students): 1450
Total cost: about € 1934
Total cost for Lab services (including template VM, Files share and public IP): about € 2100
The feedback we received from the instructor was very positive. From an admin point of view, we think the initial setup was fairly easy. However, the challenges that were discussed above introduced some extra configuration and testing efforts for admins as well as the instructors.
"Azure Lab student VM service made it possible to run Deep Learning with Audio course without any interruption, this is because the course content requires GPU resources to train several deep learning models. Students were able to use their own dataset and explore generating their own content based on these trained models. Using VM service was a great benefit for the Deep Learning with Audio course" - Koray Tahiroglu, lecturer and head of research group
Azure Lab Services introduced some major updates (April 2022) which addresses some of the shortcomings of the service and brings better visibility to student virtual machines and cost management.