REDIRECT TRAFFIC : DOUBLE TUNNEL SSH

REDIRECT TRAFFIC : DOUBLE TUNNEL SSH


ssh -L PORT1:HOST2:PORT2 -N HOST1

It will listen connection on localhost:port1 redirect traffic through host1 (so you will need an account on host 1 -but not on host2-) and finally redirect traffic to host2:port2

example
from your localhost:9999 through 192.168.0.50 to 192.168.0.70:32400

ssh -L 9999:192.168.0.70:32400 -N 192.168.0.50

Other more secured methods : http://superuser.com/a/97007/486518
French tutorial : http://www.linux-france.org/prj/edu/archinet/systeme/ch13s04.html

What is MSys - MinGW - Cygwin

To make the milky 'soup' of project names more clear:

• Cygwin - a Linux like environment, it provides tools and executable built upon cygwin dll
• MinGW - is a minimalist development environment for native Microsoft Windows applications (think: GCC to compile native Win32 applications). This is a C/C++ compiler suite which allows you to create Windows executables (without dependency on DLLs such Cygwin DLL - you only need the normal MSVC runtime)
• MSYS - is a minimal POSIX emulation layer providing a Bourne Shell command line interpreter system. It's a perfect environment for MinGW. In the past, it was forked from Cygwin
• MSys2 - is another fork of Cygwin with the same idea as MSYS, but it is kept up-to-date with Cygwin and it comes with a package management system called Pacman(calling pacman -Syu will update all of the installed packages to their newest versions). MSys2 is the basis for the upcoming successor of msysGit.
• msysGit - is the name of this project, a build environment to develop (i.e. not to use) Git for Windows, which releases the official binaries

So,

MinGW-w64 is an up to date MinGW version ===> pick it if you need a compiler

MSys2 is an up to date MSYS version (a POSIX env) ===> pick it if you need an env
                    AND a POSIX env for MinGW-w64 and others
                    AND a package manager for itself and MinGW-w64 (or others?)


https://github.com/msysgit/msysgit/wiki 
http://stackoverflow.com/questions/771756/what-is-the-difference-between-cygwin-and-mingw

apache spark vs apache hadoop map reduce

SOURCE : http://www.quora.com/What-is-the-difference-between-Apache-Spark-and-Apache-Hadoop-Map-Reduce
Suman Bharadwaj

48 upvotes by Aravindan Balan, Christophe Cop, Fangbo Tao, (more)

I'll mention the differences present at the shuffle side at a very high level, as I understand it, between Apache Spark and Apache Hadoop Map reduce. 

Since few folks have already mentioned about difference in terms of I/O etc, I'll stick to only the shuffle difference.

First Let's discuss the Map side differences:

Map Side of Hadoop Map Reduce ( see left side of the image above ): 

• Each Map task outputs the data in Key and Value pair.
• The output is stored in a CIRCULAR BUFFER instead of writing to disk.
• The size of the circular buffer is around 100 MB. If the circular buffer is 80% full by default, then the data will be spilled to disk, which are called shuffle spill files.
• On a particular node, many map tasks are run as a result many spill files are created. Hadoop merges all the spill files, on a particular node, into one big file which is SORTED and PARTITIONED based on number of reducers.

Map side of Spark ( see right side of the image above)

• Initial Design:
  
  • The output of map side is written to OS BUFFER CACHE.
  • The operating system will decide if the data can stay in OS buffer cache or should it be spilled to DISK.
  • Each map task creates as many shuffle spill files as number of reducers.
  • SPARK doesn't merge and partition shuffle spill files into one big file, which is the case with Apache Hadoop.
  • Example: If there are 6000 (R) reducers and 2000 (M) map tasks, there will be (M*R) 6000*2000=12 million shuffle files. This is because, in spark, each map task creates as many shuffle spill files as number of reducers. This caused performance degradation.
  • This was the initial design of Apache Spark.
    
• Shuffle File Consolidation:
  
  • Same as above.
  • The only difference being, the map tasks which run on the same cores will be consolidated into a single file. So, each CORE will output as many shuffle files as number of reducers.
  • Example: If there are 6000(R) and 4 (C) Cores, the number of shuffle files will be (R*C) 6000*4=8000 shuffle files. Note the huge change in the number of shuffle files.

Now to the reduce side differences:

• Reduce side of Hadoop MR:
  
  • Reducer PULLS the intermediate files(shuffle files) created at the map side. And the data is loaded into memory.
  • If the buffer reaches 70% of its limit, then the data will be spilled to disk.
  • Then the spills are merged to form bigger files.
  • Finally the reduce method gets invoked.
    
• Reduce side of Apache Spark:
  
  
  • Map side PUSHES the intermediate files(shuffle files) to Reducer.
  • The data is directly written to memory.
  • If the data doesn't fit in-memory, it will be spilled to disk from spark 0.9 on-wards. Before that, an OOM(out of memory) exception would be thrown.
  • Finally the reducer functionality gets invoked.

Horton Works 2.1 Customisation

login:root
password:hadoop

Some tools

yum install nano



Set french environment

Change

nano /etc/sysconfig/keyboard

with

KEYTABLE=”fr-latin1″
MODEL=pc105+inet″
LAYOUT=”fr”
KEYBOARDTYPE=”pc”

(All available options can be found in subdirectories under /lib/kbd/keymaps/i386.)


Change

nano /root/.bashrc

with

LANG=fr_FR.ISO-8859-1
LC_CTYPE="fr_FR.ISO-8859-1"
LC_NUMERIC="fr_FR.ISO-8859-1"
LC_TIME="fr_FR.ISO-8859-1"
LC_COLLATE="fr_FR.ISO-8859-1"
LC_MONETARY="fr_FR.ISO-8859-1"
LC_MESSAGES="fr_FR.ISO-8859-1"
LC_PAPER="fr_FR.ISO-8859-1"
LC_NAME="fr_FR.ISO-8859-1"
LC_ADDRESS="fr_FR.ISO-8859-1"
LC_TELEPHONE="fr_FR.ISO-8859-1"
LC_MEASUREMENT="fr_FR.ISO-8859-1"
LC_IDENTIFICATION="fr_FR.ISO-8859-1"
LC_ALL=

if [ -n "$SSH_CLIENT" ]; then
       export LANG="fr_FR.ISO-8859-1"
fi

Cross compiling Go app

Cross compiling Go app

First build cross compiling toolchain

cd go/src
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 ./make.bash  --no-clean
CGO_ENABLED=0 GOOS=windows GOARCH=amd64 ./make.bash --no-clean

Second build your file

GOOS=windows GOARCH=386 go build -o appname.exe appname.go
GOOS=linux GOARCH=386 CGO_ENABLED=0 go build -o appname.linux appname.go


CGO_ENABLED=0 will deactivate use of CGo. So it will be a portable app. But if your app use CGo (for example by using go package "net"), it will not build.

Option : use gonative

If you want to use CGo and do cross compiling, use gonative which download each binary distribution for each platform and build cross compiling toolchain with theses binaries, so you can use CGo.

https://github.com/inconshreveable/gonative
https://inconshreveable.com/04-30-2014/cross-compiling-golang-programs-with-native-libraries/

And then use gox https://github.com/mitchellh/gox
to cross compile your application

Some links

http://www.goinggo.net/2013/10/cross-compile-your-go-programs.html
http://solovyov.net/en/2012/cross-compiling-go/
http://stackoverflow.com/questions/12168873/cross-compile-go-on-osx

Call powershell command in batch script

powershell -Command "& { 1..10 | foreach-object { "Current output:"; $_; } }"

https://dmitrysotnikov.wordpress.com/2008/06/27/powershell-script-in-a-bat-file/

Dictionary of web dev ecosystem and javascript galaxy - nodejs, angularjs, mean, express, mongodb and so on ...

client side javascript and tools

AngularJS
javascript framework - app oriented

AngularJS based starter kit/template/generator
http://www.dancancro.com/comparison-of-angularjs-application-starters/
angular-seed - template project for angular
angular-app - example project for angular
ng-boilerplate - template project for angular
generator-angular - based on yeoman
generator-angular-fullstack - based on yeoman

Backbone.js

javascript library/toolkit - app oriented

Bootstrap
javascript framework - UI responsive design oriented

Bower
dependencies manager for webapp - manage js dependencies or asset - dependencies are deployed in flat way (opposite of npm which deploy each dependency nested) - use bower.json and .bowerrc

Compass
CSS framework based on Sass

Dojo
javascript framework - app oriented

Ember.js

javascript framework - app oriented

HTML5 Boilerplate
Web project template

jQuery
javascript framework - UI oriented

Less
CSS derivated language

MooTools
javascript library - UI oriented

Sass

CSS derivated language

Stylus
CSS derivated language

Underscore.js

javascript library

Server side javascript and tools

Node.js
server-side javascript

npm
Node package manager - installed with nodejs - each dependency is deployed in a nested way (opposite of bower which deploy each dependency in a flat way) - use package.json

Express
server-side javascript framework

Jade
Javascript template engine

Swig
Javascript template engine

Other

MEAN
conceptual javascript stack - MongoDB, Express, Angular, Nodejs
Different mean stack implementation : http://meanjs.org http://mean.io
mean.js is a fork of mean.io - difference between them

Yeoman
project generator

MongoDB
document oriented database

Redis
key/value store

CoffeeScript
javascript derivated language

TypeScript
javascript derivated language

Grunt
javascript task manager/build tool - like maven or ant - use Gruntfile.js

Gulp
http://gulpjs.com/
javascript task manager/build tool - use gulpfile.js

Use npm as build tool
http://blog.keithcirkel.co.uk/how-to-use-npm-as-a-build-tool/

JSLint
javascript code quality tool

JSHint
javascript code quality tool - fork of JSLint

JSFiddle
online javascript sandbox

RequireJS
javascript module loader

Karma (formerly Testacular)
Test runner

Jasmine
Javascript test framework

Mocha
Javascript test framework

HTTP Archive
Web technology statistic

BuiltWith
Web technology statistic