Spark 啟動流程追蹤隨筆

SparkContext 內 YARN 的啟動流程
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkContext.scala



 case "yarn-standalone" | "yarn-cluster" =>
        if (master == "yarn-standalone") {
          logWarning(
            "\"yarn-standalone\" is deprecated as of Spark 1.0. Use \"yarn-cluster\" instead.")
        }
        val scheduler = try {
          val clazz = Class.forName("org.apache.spark.scheduler.cluster.YarnClusterScheduler")
          val cons = clazz.getConstructor(classOf[SparkContext])
          cons.newInstance(sc).asInstanceOf[TaskSchedulerImpl]
        } catch {
          // TODO: Enumerate the exact reasons why it can fail
          // But irrespective of it, it means we cannot proceed !
          case e: Exception => {
            throw new SparkException("YARN mode not available ?", e)
          }
        }
        val backend = try {
          val clazz =
            Class.forName("org.apache.spark.scheduler.cluster.YarnClusterSchedulerBackend")
          val cons = clazz.getConstructor(classOf[TaskSchedulerImpl], classOf[SparkContext])
          cons.newInstance(scheduler, sc).asInstanceOf[CoarseGrainedSchedulerBackend]
        } catch {
          case e: Exception => {
            throw new SparkException("YARN mode not available ?", e)
          }
        }
        scheduler.initialize(backend)
        (backend, scheduler)


     case "yarn-client" =>
        val scheduler = try {
          val clazz =
            Class.forName("org.apache.spark.scheduler.cluster.YarnClientClusterScheduler")
          val cons = clazz.getConstructor(classOf[SparkContext])
          cons.newInstance(sc).asInstanceOf[TaskSchedulerImpl]

        } catch {
          case e: Exception => {
            throw new SparkException("YARN mode not available ?", e)
          }
        }

        val backend = try {
          val clazz =
            Class.forName("org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend")
          val cons = clazz.getConstructor(classOf[TaskSchedulerImpl], classOf[SparkContext])
          cons.newInstance(scheduler, sc).asInstanceOf[CoarseGrainedSchedulerBackend]
        } catch {
          case e: Exception => {
            throw new SparkException("YARN mode not available ?", e)
          }
        }

YarnClusterScheduler
YarnClusterSchedulerBackend
YarnClientClusterScheduler
YarnClientSchedulerBackend
source code 位於
https://github.com/apache/spark/tree/master/yarn/src/main/scala/org/apache/spark/scheduler/cluster

Battle Isle 1 (1991,DOS) PART 11-15 Walkthrough 狂島浴血 11-15 關 (過關實錄)

狂島浴血 第十一關-第十五關 過關實錄

Part 11. VALEY 第十一關 總時間(time) 26:58

Part 12. TESTY 第十二關 總時間(time) 47:25

Part 13. TERRA 第十三關 總時間(time) 57:05

Part 14. SLAVE 第十四關 總時間(time) 1:02:47

Part 15. NEVER 第十五關 總時間(time) 42:28

Battle Isle 1 (1991,DOS) PART 6-10 Walkthrough 狂島浴血 6-10 關 (過關實錄)

狂島浴血 第六關-第十關 過關實錄

Part 6. RUSTY 第六關 總時間(time) 24:22

Part 7. FIFTH 第七關 總時間(time) 35:29

Part 8. VESUV 第八關 總時間(time) 32:29

Part 9. MAGIC 第九關 總時間(time) 44:11

Part 10. SPACE 第十關 總時間(time) 39:12

Battle Isle 1 (1991,DOS) PART 1-5 Walkthrough 狂島浴血 1-5 關 (過關實錄)

狂島浴血過關紀錄實況
Battle Isle 1 - 1991

Part 1. CONRA  第一關 總時間(time) 5:41

Part 2. PHASE 第二關 總時間(time) 7:44

Part 3. EXOTY 第三關 總時間(time) 9:02

Part 4. MOUNT 第四關 總時間(time) 18:54

Part 5. FIGHT 第五關 總時間(time) 20:46

sscanf with Android NDK

It seems that there are some bug in sscanf() with Android NDK-r9.

When read a line from a text file, i use the strtok() to tokenize the string, and call sscanf() to convert the token into hex value. There are some chance to get 0 from sscanf() , although it is not always occurred.

To solve the problem in NDK, i write a simple function to convert the token into hex,  the code is as follow listing:

// sscanf has bug in Android NDK
static unsigned char convertToHex( char c ) {
    if ( c >= '0' && c <= '9' ) {
        return c - '0';
    }
    if ( c >= 'A' && c <= 'F' ) {
        return c - 'A' + 10;
    }
    if ( c >= 'a' && c <= 'f' ) {
        return c - 'a' + 10;
    }
    return 0;
}
static unsigned char bytesToHex( char *token ) {
    unsigned char tmp ;
    unsigned char tmp1;
    unsigned char tmp2;
    tmp1 = convertToHex( token[0] );
    tmp2 = convertToHex( token[1] );
    tmp = tmp1 << 4 | tmp2;
    return tmp ;
};

AES-CTR-128 with OpenSSL on Android and Windows

I have some OpenSSL experience during Content Protection Project development. In this post, there some non-confidential technical skill could be shared to you.

For Developing the Content Encryption, the AES-CTR-128 was used.

To use the AES-CTR-128 easily with OpenSSL 0.9.8e
Copy aes_core.c and aes_ctr.c and related header files from OpenSSL source.

And the code sinppet should look like this:

#include

AES_key aes_key
input         : in_blk
output      : out_blk

unsigned char *key = "1234567812345678"
unsigned char nonce_counter[16];
unsigned char stream_block[16];
size_t tmp = 0;
int size = 0;

set up the AES key with AES-128
AES_set_encrypt_key(key, 128, &aes_key);

use AES-CTR-128 algorithm to encrypt content
AES_ctr128_encrypt(
   in_blk,
   out_blk,
   size,
  &aes_key,
  nonce_counter,
  stream_block,
  &tmp);


the decryption code snippet:
AES_ctr128_encrypt(
out_blk,
dec_out_blk,
size,
&aes_key,
nonce_counter,
stream_block, &tmp);


android_ndk demo 

visual_studio_2010_demo

Chrome Global Dark Style

一個 Chrome 的 Plugin 可以讓網頁瀏覽變成黑底白字
http://userstyles.org/styles/23516/midnight-surfing-global-dark-style