| Author | Message | ||
Kindal Side Hero Username: Kindal Post Number: 3604 Registered: 09-2007 Posted From: 14.139.128.20 Rating: N/A |
Difficult question, recently spoke to a data scientist. He said, their company (in India), hired consultants to set up Hadoop ecosystem and trained a guy to handle all issues related to Hadoop. He also said, anyone working on Hadoop requires holistic knowledge on Hadoop. This is a good article describing the skill sets required for Hadoop admin. So what skills do Hadoop Administrators need? First and foremost, Hadoop admins need general operational expertise such as good troubleshooting skills, understanding of system’s capacity, bottlenecks, basics of memory, CPU, OS, storage, and networks. I will assume that any good DBA has these covered. Second, good knowledge of Linux is required, especially for DBAs who spent their life working with Solaris, AIX, and HPUX. Hadoop runs on Linux. They need to learn Linux security, configuration, tuning, troubleshooting, and monitoring. Familiarity with open source configuration management and deployment tools such as Puppet or Chef can help. Linux scripting (perl / bash) is also important – they will need to build a lot of their own tools here. Third, they need Hadoop skills. There’s no way to avoid this  They need to be able to deploy Hadoop cluster, add and remove nodes, figure out why a job is stuck or failing, configure and tune the cluster, find the bottlenecks, monitor critical parts of the cluster, configure name-node high availability, pick a scheduler and configure it to meet SLAs, and sometimes even take backups. So yes, there’s a lot to learn. But very little of it is Java, and there is no reason DBAs can’t do it. However, with Hadoop Administrator being one of the hottest jobs in the market (judging by my LinkedIn inbox), they may not stay DBAs for long after they become Hadoop Admins… Any DBAs out there training to become Hadoop admins? Agree that Java isn’t that important? Let me know in the comments. - See more at: http://www.pythian.com/blog/hadoop-faq-but-what-about-the-db as/#sthash.B69Lr7xT.dpuf 5 mins after your birth, they decide your name, nationality, religion & sect & you spend rest of ur life defending something u didn't choose. | ||
Bumper Side Hero Username: Bumper Post Number: 7266 Registered: 07-2014 Posted From: 199.168.151.87 Rating: N/A |
thanks vnkle Snakes in Your backyard won't bite only your neighbours. #Hillary Clinton | ||
Kindal Side Hero Username: Kindal Post Number: 3603 Registered: 09-2007 Posted From: 14.139.128.20 Rating: N/A |
A good article that clears a lot of concerns on Spark is http://thomaswdinsmore.com/2015/04/21/spark-too-big-to-fail/ Here is the list of Companies & Organizations adopting Spark https://cwiki.apache.org/confluence/display/SPARK/Powered+By +Spark 5 mins after your birth, they decide your name, nationality, religion & sect & you spend rest of ur life defending something u didn't choose. | ||
Gatti_gunde Megastar Username: Gatti_gunde Post Number: 20676 Registered: 02-2010 Posted From: 204.194.77.3 Rating: N/A |
Hadoop administrators em nerchukovaali skills better avvali ante? Boob annai ni kindhesi thanthe DB set aithadhi | ||
Kindal Side Hero Username: Kindal Post Number: 3600 Registered: 09-2007 Posted From: 14.139.128.20 Rating: N/A |
Important note. There is a lot of debate on whether Spark will catch up and live upto the expectations. Browse, and read 3-5 articles. Don't jump into any conclusions. If you are serious about switching towards Bigdata, the time spent in learning Spark will definitely help you in the short-term & long-term. Without downplaying the criticism, Spark, as-is has huge potential, and is here to stay. 5 mins after your birth, they decide your name, nationality, religion & sect & you spend rest of ur life defending something u didn't choose. | ||
Kindal Side Hero Username: Kindal Post Number: 3599 Registered: 09-2007 Posted From: 14.139.128.20 Rating: N/A |
Will be posting useful links on Spark. Begin here: take your time, spend sometime, take notes and get the grasp of where Spark is used in Bigdata https://dzone.com/refcardz/apache-spark 5 mins after your birth, they decide your name, nationality, religion & sect & you spend rest of ur life defending something u didn't choose. |