In case of authentication errors related to authenticated hosts after re-launching your VM you can regenerate your known_hosts file by removing it and SSHing into the machine again: $ cd ~/.ssh/ $ rm -f known_hosts $ ssh -i ~/.ssh/filename you want to run your Python project as a package or for importing you can install the local code through pip on the VM: $ cd path/to/code/folder $ python3 -m pip install -e. For example: alias gcp_ssh = 'ssh -i ~/.ssh/filename alias port_forward = 'ssh -i ~/.ssh/filename -N -L localhost:8888:localhost:8888 alias give_me_the_power = 'sudo chown -R gcp_username:gcp_username path/to/code/folder' You can make life easier for yourself and add some aliases to your ~/.bashrc file to speed up the terminal commands if you want to do this more often. You can enable file-syncing on save which will reflect all the changes in your code from local to the VM in real-time for live development. Go to Preferences → Packages → Hydrogen → Settings and add the following to the Kernel Gateways field: [ Open Atom and install the Hydrogen package. In a new local terminal window write: ssh -i ~/.ssh/filename -N -L localhost:8888:localhost:8888 my case no messages show up in the terminal which is perhaps a bit confusing, just keep the terminal window open. This can be achieved by creating a SSH tunnel. To communicate with the Jupyter kernel running on the VM we use port forwarding from our local machine to the VM. Also, no monitor means we don’t need to launch a browser in the VM ) $ jupyter-notebook -no-browser -port=8888 While connected to the VM in a terminal launch a Jupyter kernel on a port of your choice, I use 8888. We can check if the connection and authentication is okay so far by SSHing into the Virtual Machine from a local terminal: $ ssh -i ~/.ssh/place_of_ssh_keys Create Jupyter Kernel & SSH tunnel ![]() Don’t forget to click save all the way in the bottom of the page. Click the name of your VM → Edit → Network Interfaces → External IP. To access the VM with an external SSH connection we need to assign it an external IP address. Click edit and paste your public SSH key in the field, save it. Navigate to your VM: Compute Engine → VM Instances → MetaData → SSH Keys. On your local machine open the browser and go to the GCP console. Next, make sure the VM accepts authentication. Spin up the VM with your bash script or terminal commands and ensure Jupyter is installed on the machine: $ pip install jupyter You can view the generated public SSH key with: $ cat ~/.ssh/filename.pub Make sure to place it somewhere logical (like ~/.ssh), give it a filename and think of a passphrase. On a Mac we can create an SSH keypair with the included ssh-keygen through the terminal: $ cd ~/.ssh/ $ ssh-keygen -m PEM -t rsa -C "GCP_username" To communicate with a Virtual Machine we need some form of authentication. If your weapon of choice is GCP with RStudio, P圜harm (or another JetBrains product) your life is a bit easier. There should be many of similarities in the steps below for alternative set ups. In this example, I will be developing in Python using my favorite hackable editor Atom powered by Hydrogen running in a machine on Google Cloud Platform (GCP). I decided to document these steps for my future self and hope someone else will find it useful. ![]() I quickly discovered RStudio and P圜ham Professional both have options for remote development and figured how hard could it be? Yet, I was not able to find a clear end-to-end write-up of the required bits and pieces. This leads to the simple wish to work in an Integrated Development Environment (IDE) of choice while using the computational power and resources of a Virtual Machine. ![]() More specifically, robust and reproducible Machine Learning products or experiments require a better foundation for writing production-grade code than a notebook. Notebooks are great for a quick one-time analysis but does not offer much in terms of software engineering best practices. A straightforward solution is to work in your browser through a Jupyter or Google Colab Notebook connected to the remote machine. However, remote development and live experimenting with code changes within a VM requires a couple steps. Once you have written some code, executing it on a Virtual Machine (VM) for one-time results or scheduled production purposes is relatively easy. Photo by israel palacio on Unsplash IntroĪnyone interested in modelling large amounts of data or doing some kind of Deep Learning work will quickly reach the limits of their local machine.
0 Comments
Leave a Reply. |