ZHIYUAN (ZOE) LIN
  • Home
  • Academic Notes
  • Thoughts
  • Chinese

Launch EC2 Spark Cluster

12/5/2015

1 Comment

 
(Spark maintains a document about this, but I find it's a bit difficult to understand if you are new to Spark. ) Here is a simpler guide (at least for me). 
1. Log in to your ASW (EC2 Console) account, click your username on the top-right corner, and switch to "Security Credentials". You can see your  Access Keys (Access Key ID and Secret Access Key). Create one and download it if you don't have one. Then export your Access Keys in terminal. e.g.,         
export AWS_ACCESS_KEY_ID=ABCDEFG1234567890123
export AWS_SECRET_ACCESS_KEY=AaBbCcDdEeFGgHhIiJjKkLlMmNnOoPpQqRrSsTtU 
2. Prepare the key file. You can create one and download it on EC2 (usually ends with .pem). Then change its mod. 
chmod 600 zhiyuanKey.pem 
3. Go to Spark/ec2 directory. Run the launch command: 
​ ./spark-ec2 -k <keypair> -i <key-file> -s <num-slaves> launch <cluster-name>
For example: (this also specifies instance type,  custom AMI, and AMI region)
./spark-ec2 -k zhiyuanKey -i zhiyuanKey.pem -s 4 -t g2.2xlarge -a ami-71280941 --region=us-west-2 --zone=us-west-2a launch zhiyuanCluster

​

Now, you are all done!!
Login the master node:  (you can either login as the root, or as the ec2-user)
./spark-ec2 -k <keypair> -i <key-file> login <cluster-name>
ssh -i <key-file> ec2-user@machine_ip    // (if you are using ubuntu, make sure you have changed "c2-user" to "ubuntu") 

Stop Cluster
./spark-ec2 --region=<ec2-region> stop <cluster-name>
Restart Cluster
./spark-ec2 -i <key-file> --region=<ec2-region> start <cluster-name>
Destroy Cluster
./spark-ec2 --region=<ec2-region> destroy <cluster-name>
1 Comment
Commercial Cleaning Illinois link
12/5/2022 12:08:33

Great post, thanks for writing

Reply



Leave a Reply.

    Categories

    All
    Artificial Intelligence
    GPU Programming
    Paper Review
    Scala
    Spark

    Archives

    December 2015
    November 2015
    October 2015
    September 2015
    August 2015

    RSS Feed

                                                                                                                   © 2015 Zhiyuan Lin Reserved