REDIRECT TRAFFIC : DOUBLE TUNNEL SSH

REDIRECT TRAFFIC : DOUBLE TUNNEL SSH


ssh -L PORT1:HOST2:PORT2 -N HOST1

It will listen connection on localhost:port1 redirect traffic through host1 (so you will need an account on host 1 -but not on host2-) and finally redirect traffic to host2:port2

example
from your localhost:9999 through 192.168.0.50 to 192.168.0.70:32400

ssh -L 9999:192.168.0.70:32400 -N 192.168.0.50

Other more secured methods : http://superuser.com/a/97007/486518
French tutorial : http://www.linux-france.org/prj/edu/archinet/systeme/ch13s04.html

What is MSys - MinGW - Cygwin

To make the milky 'soup' of project names more clear:

• Cygwin - a Linux like environment, it provides tools and executable built upon cygwin dll
• MinGW - is a minimalist development environment for native Microsoft Windows applications (think: GCC to compile native Win32 applications). This is a C/C++ compiler suite which allows you to create Windows executables (without dependency on DLLs such Cygwin DLL - you only need the normal MSVC runtime)
• MSYS - is a minimal POSIX emulation layer providing a Bourne Shell command line interpreter system. It's a perfect environment for MinGW. In the past, it was forked from Cygwin
• MSys2 - is another fork of Cygwin with the same idea as MSYS, but it is kept up-to-date with Cygwin and it comes with a package management system called Pacman(calling pacman -Syu will update all of the installed packages to their newest versions). MSys2 is the basis for the upcoming successor of msysGit.
• msysGit - is the name of this project, a build environment to develop (i.e. not to use) Git for Windows, which releases the official binaries

So,

MinGW-w64 is an up to date MinGW version ===> pick it if you need a compiler

MSys2 is an up to date MSYS version (a POSIX env) ===> pick it if you need an env
                    AND a POSIX env for MinGW-w64 and others
                    AND a package manager for itself and MinGW-w64 (or others?)


https://github.com/msysgit/msysgit/wiki 
http://stackoverflow.com/questions/771756/what-is-the-difference-between-cygwin-and-mingw

apache spark vs apache hadoop map reduce

SOURCE : http://www.quora.com/What-is-the-difference-between-Apache-Spark-and-Apache-Hadoop-Map-Reduce
Suman Bharadwaj

48 upvotes by Aravindan Balan, Christophe Cop, Fangbo Tao, (more)

I'll mention the differences present at the shuffle side at a very high level, as I understand it, between Apache Spark and Apache Hadoop Map reduce. 

Since few folks have already mentioned about difference in terms of I/O etc, I'll stick to only the shuffle difference.

First Let's discuss the Map side differences:

Map Side of Hadoop Map Reduce ( see left side of the image above ): 

• Each Map task outputs the data in Key and Value pair.
• The output is stored in a CIRCULAR BUFFER instead of writing to disk.
• The size of the circular buffer is around 100 MB. If the circular buffer is 80% full by default, then the data will be spilled to disk, which are called shuffle spill files.
• On a particular node, many map tasks are run as a result many spill files are created. Hadoop merges all the spill files, on a particular node, into one big file which is SORTED and PARTITIONED based on number of reducers.

Map side of Spark ( see right side of the image above)

• Initial Design:
  
  • The output of map side is written to OS BUFFER CACHE.
  • The operating system will decide if the data can stay in OS buffer cache or should it be spilled to DISK.
  • Each map task creates as many shuffle spill files as number of reducers.
  • SPARK doesn't merge and partition shuffle spill files into one big file, which is the case with Apache Hadoop.
  • Example: If there are 6000 (R) reducers and 2000 (M) map tasks, there will be (M*R) 6000*2000=12 million shuffle files. This is because, in spark, each map task creates as many shuffle spill files as number of reducers. This caused performance degradation.
  • This was the initial design of Apache Spark.
    
• Shuffle File Consolidation:
  
  • Same as above.
  • The only difference being, the map tasks which run on the same cores will be consolidated into a single file. So, each CORE will output as many shuffle files as number of reducers.
  • Example: If there are 6000(R) and 4 (C) Cores, the number of shuffle files will be (R*C) 6000*4=8000 shuffle files. Note the huge change in the number of shuffle files.

Now to the reduce side differences:

• Reduce side of Hadoop MR:
  
  • Reducer PULLS the intermediate files(shuffle files) created at the map side. And the data is loaded into memory.
  • If the buffer reaches 70% of its limit, then the data will be spilled to disk.
  • Then the spills are merged to form bigger files.
  • Finally the reduce method gets invoked.
    
• Reduce side of Apache Spark:
  
  
  • Map side PUSHES the intermediate files(shuffle files) to Reducer.
  • The data is directly written to memory.
  • If the data doesn't fit in-memory, it will be spilled to disk from spark 0.9 on-wards. Before that, an OOM(out of memory) exception would be thrown.
  • Finally the reducer functionality gets invoked.

Horton Works 2.1 Customisation

login:root
password:hadoop

Some tools

yum install nano



Set french environment

Change

nano /etc/sysconfig/keyboard

with

KEYTABLE=”fr-latin1″
MODEL=pc105+inet″
LAYOUT=”fr”
KEYBOARDTYPE=”pc”

(All available options can be found in subdirectories under /lib/kbd/keymaps/i386.)


Change

nano /root/.bashrc

with

LANG=fr_FR.ISO-8859-1
LC_CTYPE="fr_FR.ISO-8859-1"
LC_NUMERIC="fr_FR.ISO-8859-1"
LC_TIME="fr_FR.ISO-8859-1"
LC_COLLATE="fr_FR.ISO-8859-1"
LC_MONETARY="fr_FR.ISO-8859-1"
LC_MESSAGES="fr_FR.ISO-8859-1"
LC_PAPER="fr_FR.ISO-8859-1"
LC_NAME="fr_FR.ISO-8859-1"
LC_ADDRESS="fr_FR.ISO-8859-1"
LC_TELEPHONE="fr_FR.ISO-8859-1"
LC_MEASUREMENT="fr_FR.ISO-8859-1"
LC_IDENTIFICATION="fr_FR.ISO-8859-1"
LC_ALL=

if [ -n "$SSH_CLIENT" ]; then
       export LANG="fr_FR.ISO-8859-1"
fi

Cross compiling Go app

Cross compiling Go app

First build cross compiling toolchain

cd go/src
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 ./make.bash  --no-clean
CGO_ENABLED=0 GOOS=windows GOARCH=amd64 ./make.bash --no-clean

Second build your file

GOOS=windows GOARCH=386 go build -o appname.exe appname.go
GOOS=linux GOARCH=386 CGO_ENABLED=0 go build -o appname.linux appname.go


CGO_ENABLED=0 will deactivate use of CGo. So it will be a portable app. But if your app use CGo (for example by using go package "net"), it will not build.

Option : use gonative

If you want to use CGo and do cross compiling, use gonative which download each binary distribution for each platform and build cross compiling toolchain with theses binaries, so you can use CGo.

https://github.com/inconshreveable/gonative
https://inconshreveable.com/04-30-2014/cross-compiling-golang-programs-with-native-libraries/

And then use gox https://github.com/mitchellh/gox
to cross compile your application

Some links

http://www.goinggo.net/2013/10/cross-compile-your-go-programs.html
http://solovyov.net/en/2012/cross-compiling-go/
http://stackoverflow.com/questions/12168873/cross-compile-go-on-osx

Call powershell command in batch script

powershell -Command "& { 1..10 | foreach-object { "Current output:"; $_; } }"

https://dmitrysotnikov.wordpress.com/2008/06/27/powershell-script-in-a-bat-file/