Redis Backup & Restore

Backup KEY

redis-cli --raw dump your.data.key.name > dump.rdb

Restore KEY

cat dump.rdb | head -c-1 | redis-cli -x restore your.data.key.name 0

 

Advertisements

Linux 虛擬化打機心得 Part 1

長久以來,遊戲市場對於 Linux 平台絕對是沒什麼競爭力。原生遊戲稀少之餘很多時都沒人會玩,一般遊戲開發商對個人電腦市場亦只會推出 Windows 或是 Mac 版本。因此有一段很長時間, Linux 使用者主要靠 Wine 程式模擬 Windows 環境來玩 Windows 遊戲。

近年發展可能漸趨成熟,遊戲產業出現些微變化。例如以 Wine 為底層的 PlayOnLinux 提供一鍵安裝介面簡化玩 Windows 遊戲的難度。知名遊戲平台 Stream 亦提供 Linux 客戶端;除此以外更進一步推出 Linux 發行版 StreamOS,為改善開源遊戲市場做出不少貢獻。

由於先天性的技術限制 ,Wine 效能不佳,能成功運行的遊戲並不多。Wine 開發進度緩慢目前只支援到 DirectX 9.0 c ,意味著過去數年推出的遊戲大部分都玩不了。 Value 在 Steam 上的努力確實帶動不少社群投入開源遊戲移植與開發。但相關工作有不少難度且非一朝一夕之事,要令全數遊戲支援 Linux 系統更是天方夜譚。

筆者主要使用 Linux 系統多年,工作上以至生活習慣經已無法脫離。解決辦法例如 Dual Boot 缺點是不能同時間享受兩個系統帶來的好處,另外再買一台電腦玩遊戲會太花錢又浪費空間。剛好最近家中多出一張顯示卡,如果花點時間弄弄就能在原有的 Linux 系統上玩不同的主流遊戲,Why not?

若果想法相同,下面的心得或者會適合你。

VGA Passthrough

VGA Passthrough 並非新事物,技術已經存在多年。簡單來說就是透過虛擬化技術把連接了顯示卡的插槽穿透虛擬層,直接將物理設置連接到虛擬機 (PCI Passthrough) ,實現與裸機系統相比接近零差距的效能表現。

再簡單一句話就是,在 Linux 系統上設一台 Windows 虛擬機打遊戲。

Linux 虛擬化技術

1_nFUPIlIUkFwETptGO4iFvA.png
KVM 架構圖

KVM (Kernel-Based Virtual Machine) 是一個 Linux 內核模組提供硬件全虛擬化,結合 QEMU 模擬器組成全虛擬機技術,利用 libvirt 進行虛擬設備管理。

以下是筆者的電腦規格,設備上可能有點舊,不過操作上絕無區別。唯一需要注意是使用新舊版 Ubuntu 指令是否會有不同。下面的內容主要是針對 Intel / NVIDIA 架構而設,除此以外的配備(如 AMD/ATI )在設定上會有些微不同,那部分就留待大家自行尋找。

Software
Host OS: Ubuntu 16.04.5 LTS
Hardware
CPU: Intel® Xeon(R) CPU E3–1245 V2 @ 3.40GHz × 8
Motherboard: Gigabyte GA-H77N-WIFI (LGA 1155, Intel® h77 chipset)
Memory: Kingston 16GB (2x8GB DDR3 1600Mhz)
Graphic: Gigabyte NVIDIA GTX 1050 Ti

硬件檢測

實現全虛擬化技術需要硬件支援,並非任何一台電腦都適合的,基本上查一下 CPU 和 主機板的規格就能斷定 。以 Intel 架構來說我們需要確認 VT-x VT-d 兩者有否支援。如果是知道處理器型號的話,比較簡單的就是到官方網站看規格:https://ark.intel.com/zh-tw

1_JRkKrusrd6MY3fF73raUdw.png
Intel 官方網頁會標示出 CPU 的 VT-x , VT-d 的支援狀況

至於主機板的支援情況比較模糊,還是直接在電腦上操作比較理想。

VT-x (Intel Virtualization Technology)

運行以下指令,若果回傳 vmx 則代表 CPU 支援 VT-x 技術。

> grep --color vmx /proc/cpuinfo

不過支援歸支援,能不能用有沒有打通是另一回事。

輸入下面指令,回傳 3 或 5 代表 VT-x 功能已經正常啟動:

> sudo apt-get install msr-tools -y 
> sudo modprobe msr          # load msr module
> sudo rdmsr 0x3a            # check if VT-x is enabled in UEFI

簡單 3 行指令看似複雜,背後意思其實是在查閱 CPU MSR 寄存器的值; 地址 0x3a (IA32_FEATURE_CONTROL) 控制 Intel VT-x 功能的開關。

注意 VT-x 功能必需在 BIOS 設成啟用,若果以上指令回傳其他數值不妨到這𥚃檢查一下再重試。

1__QnQncTIdhLgtWyBM2jJ3Q
BIOS VTx 設定

IOMMU

 

1_lYWkCsMmqqVPgYmFT1zTbw.png
IOMMU功能啟動後系統的運作結構

IOMMU(Input/Output Memory Management Unit)是北橋技術的一部分,實現例子有 VT-d (Intel Virtualization Technology for Directed I/O) 或是 AMD-Vi (AMD’s I/O Virtualization Technology),其作用是實踐I/O虛擬化,能夠將不同的物理I/O設備分配到虛擬機或與虛擬機群共用。上面提到的 VGA Passthrough 技巧就是透過 IOMMU 達成。

同樣地,Vt-d 功能需要在 BIOS 中啟用 ,預設是關閉的。若果你找不到這個設定,很大可能代表這張主機板沒有 Vt-d 功能。必須換一張有 Vt-d 功能的主機板才能繼續下面各部分。

1_rYSFxUAk82flWJuRHoC_tg.jpeg
BIOS VT-d 設定

在 Ubuntu 中啟用 IOMMU

打開 /etc/default/grub 尋找下面這個變數,在引號最後位置添加intel_iommu=on

GRUB_CMDLINE_LINUX_DEFAULT="…"

例子:

# 添加前
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
# 添加後
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on"

輸入 update-grub 更新 grub 設置後重啟電腦。

檢查 IOMMU 啟用狀態

網上普遍方法是查閱系統啟動訊息,筆者發現很多教學都是用下面這個方法來確認,但其實不準確甚至可以說是錯誤的。

> dmesg | grep -i iommu
DMAR: IOMMU enabled …

這個方法實際上只是在系統啟動時檢查 grub 是否帶有 “intel_iommu=” 參數,除此以外別無其它用意。正確的做法應該是這樣:

dmesg | grep "Directed I/O"
0_-48j7vu0WA_KBgp4

無法成功打開的朋友們很抱歉,這代表該設備不具備全虛擬化能力,你可以嘗試更換一下處理器或者主機板。重點是必需通過這一關才能繼續下一個部分。

開機模式

近代電腦的開機模式 (Boot Mode) 通常是用 UEFI 模式而非舊有的 Legacy BIOS。除非特別原因,否則安裝主流 Linux Distros 例如 Ubuntu,在一般情況下預設會以 UEFI 模式安裝。

驗證方法相當簡單,如果以 UEFI 啟動系統會掛載 efi 目錄,否則就是以 Legacy BIOS 啟動。用下面指令查一下該目錄是否存在:

ls /sys/firmware/efi

在接下來的內容會以 UEFI 模式進行,若發現系統用的是 Legacy BIOS,你可能需要找方法設成 UEFI。
事實上顯示卡亦需確認是否 UEFI-Ready。由於驗證方法比較麻煩,而且近年大部分的都支援 UEFI,選用主流顯示卡應該沒有問題,超過八年以上的才有需要核實一下。

上面提到的兩種啟動模式,其實都有機會達成目的。如果你的顯示卡太舊不妨嘗試以 Legacy BIOS 方式操作,設定上有些微不同且步驟較多,詳細要自行到網上搜索。

圖像處理器

要成功 VGA Passthrough,電腦設備必需要有至少 2 個圖像處理器,簡單達成方法有:

  1. 2張外置顯示卡
  2. 1張外置顯示卡,1個內置顯示的處理器

用指令確認系統是不是偵測到 2 個 VGA :

lspci | grep VGA

0_dB7y_lv8ax7Nx2ll.png
偵測出 2 個圖像控制器

(1)憑肉眼相信已經可以確認,所以上面方法是為(2)而設。這一步主要是了解正在使用的處理器是否具有內置顯示能力。除了可以從官方網站確認之外,在電腦上直接操作也行。另外這個方法對 (1) 亦適用。

如果 CPU 內置圖像處理器,Intel 官方網站會列出其規格

到這𥚃硬件檢測的部分就結束了。如果上面沒問題代表硬件已經準備就緒,剩下的都是軟件問題。

待續 –

Gmail + Postfix on Ubuntu 14.04

What if I tell you that this article has already existed for 400 days in my draft list (more than a year)? This probably makes you wonder how ridiculous I am to not finish it earlier. Okay, the fact is that I put too much content in one article. Now, I plan to break it down into a series of article.

In this article, you will learn how to setup a free Gmail Relay on your Ubuntu server. There maybe quite a bunch of online resources doing similar thing, but I prefer to write my own as an example and to refresh my forgetful brain. 😀

Installing Postfix

Install Postfix using the following command:

sudo apt-get install postfix

You will see this prompt at some point. Select Internet Site.

Screenshot from 2016-04-13 22:43:47

Screenshot from 2016-04-16 17:07:18

Postfix Configuration

Once the installation is finished, change your terminal’s current directory to

/etc/postfix

Configure Gmail account and password

create a file call gmail_passwd, replace the information with your credentials.

[smtp.gmail.com]:587 username@gmail.com:password

execute the following command to generate a postfix database.

sudo postmap /etc/postfix/gmail_passwd

For safely reason, you should allow only the root user can access this 2 files.

sudo chown root:root /etc/postfix/gmail_passwd /etc/postfix/gmail_passwd.db
sudo chmod 0600 /etc/postfix/gmail_passwd /etc/postfix/gmail_passwd.db

 Configure your Relay Server

open /etc/postfix/main.cf with whatever editor you like:

vim /etc/postfix/main.cf

add the following line to map your credentials

smtp_sasl_password_maps = hash:/etc/postfix/gmail_passwd

Enable TLS and sasl authiencation for Gmail SMTP Server

smtp_use_tls = yes
smtp_sasl_auth_enable = yes

Gmail SMTP Server does not support postfix’s security options. Reset them by:

smtp_sasl_security_options =

Relayhost field means the SMTP relay host. Please put the GMail SMTP server address here. Make sure this address is identical to the one in gmail_passwd

relayhost = [smtp.gmail.com]:587

here is the complete version of the main.cf:

# See /usr/share/postfix/main.cf.dist for a commented, more complete version

# Debian specific: Specifying a file name will cause the first
# line of that file to be used as the name. The Debian default
# is /etc/mailname.
#myorigin = /etc/mailname

smtpd_banner = $myhostname ESMTP $mail_name (Ubuntu)
biff = no

# appending .domain is the MUA's job.
append_dot_mydomain = no

# Uncomment the next line to generate "delayed mail" warnings
#delay_warning_time = 4h

readme_directory = no

# TLS parameters
smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
smtpd_use_tls=yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_use_tls = yes
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtp_sasl_password_maps = hash:/etc/postfix/gmail_passwd
smtp_sasl_auth_enable = yes
smtp_sasl_security_options =
# See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for
# information on enabling SSL in the smtp client.

smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination
myhostname = office-local
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
mydestination = office-local, localhost.localdomain, , localhost
relayhost = [smtp.gmail.com]:587
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all

reload the configuration:

service postfix reload 

check the status by:

tail -f /var/log/mail.log

check mail.err if there is any error. This file may not exist if there is no error, or it is the first time of you to configure mail related service.

tail -f /var/log/mail.err

Gmail Account Setting

Screenshot from 2016-04-16 16:34:10

Postfix may not works with the default security level of Gmail. Please turn on the “allow less secure apps” option via this link:

https://www.google.com/settings/security/lesssecureapps


 Testing

mailx is a popular utility in Linux World for sending and receiving mail in terminal. Execute the following command to install it.

sudo apt-get install mailx

Now you can test your postfix configuration by sending a mail to someone. Replace to@whatever.com with a valid email address.

echo "Hello Body" | mail -s "Hello Subject" -a to@whatever.com

If it was sent successfully, you should see this in the mail.log:

Apr 16 08:41:57 office-local postfix/smtp[1274]: 1603753E: to=<xxx@whatever.com>, relay=smtp.gmail.com[64.233.189.108]:587, delay=1723, delays=1720/0.01/1.4/0.85, dsn=2.0.0, status=sent (250 2.0.0 OK 1460796117 h5sm69972904pat.0 - gsmtp)
Apr 16 08:41:57 office-local postfix/qmgr[1197]: 1603753E: removed

 Trouble Shooting

Apr 16 08:27:23 office-local postfix/smtp[1007]: E81E93C5: to=<xxx@whatever.com>, relay=smtp.gmail.com[64.233.189.109]:587, delay=7, delays=0.01/0.01/7/0, dsn=4.7.14, status=deferred (SASL authentication failed; server smtp.gmail.com[64.233.189.109] said: 534-5.7.14 <https://accounts.google.com/signin/continue?sarp=1&scc=1&plt=AKgnsbtS?534-5.7.14 qQtdCND0kjQn0lwMS5NJc9nXMTotKJG76X83IjhjNi5jaGjyZed5WfHf_Rm0SxoaiwFAAU?534-5.7.14 tvpINwFpyp2YOKpYmEnCnlmUn33voaf7jG8lN71043p3bgcKNY8Zqk2WB-zKKEoWW58zkM?534-5.7.14 uwn5lZcHpxEJvx5HXAEf0MQcOu_EVJLEFByoP35IPcKquJM5qnI4nBVGzFD93uC6IaSxDH?534-5.7.14 DhDeKmFa94G2XQXZItNqqLtrW8Yco> Please log in via your web browser and?534-5.7.14 then try again.?534-5.7.14 Learn more at?534 5.7.14 https://support.google.com/mail/answer/78754 n74sm69735534pfa.45 - gsmtp)

your Gmail account is not allow less-secure apps to sign-in


 

Reference

http://mhawthorne.net/posts/postfix-configuring-gmail-as-relay.html

https://community.runabove.com/kb/en/instances/how-to-relay-postfix-mails-via-smtp.gmail.com-on-ubuntu-14.04.html

https://easyengine.io/tutorials/linux/ubuntu-postfix-gmail-smtp/

OneTab keeps crashing

It was just a normal day, a day similar to others.  My OneTab was suddenly not working. Some people here might not know it, OneTab is a chrome extension that makes keeping and restoring tabs easy.

Screenshot from 2016-04-06 18:04:16.png

Honestly , I didn’t get surprised. Why?

I have a habit of keeping lots of tabs open in multiple windows. In a regular day, I can open up to 30+ tabs easily. That’s why I need some tools like OneTab to manage all of them. Also, I keep all the lists even they are restored. This led to that over the past 2 years, I have 4k+ tab records inside the OneTab.

The reason is simple, when I open up the OneTab page, I have a sense that it becomes slower and slower, it’s because there are too many items in the page which may cause a memory leak. What’s worse is that I can’t even remove and export the data!

So, what can I do?

Popular Linux Distro mostly is storing the data of the chrome extensions under the following path:

 /home/{YOURNAME}/.config/google-chrome/Default/Local Storage

For each extensions, there would be a .localstorage file and a .localstorage-journal file. One tab is stored as:

 chrome-extension_chphlpgkkbolifaimnlloiipkdnihall_0.localstorage

 chrome-extension_chphlpgkkbolifaimnlloiipkdnihall_0.localstorage-journal

In fact, those .localstorage files are just SQLite files. Fortunately I can solve it by doing some DB operations.

To make life simpler, I do it in GUI. Thanks to the author of sqlitebrowser.

Screenshot from 2016-04-12 00:36:38.png

Screenshot from 2016-04-12 00:37:462.png

Basically, it’s just a simple database containing 1 key-value table only. There are 7 records in this table but only 2 records, idCounter and state, are matters.

  • idCounter: the last primary key of tabs
  • state: the tabs records that stored in JSON string

Copy and paste the value of “state” to any editor, you should see something like this:

{
  "tabGroups": [
    {
      "id": 3198,
      "tabsMeta": [
        {
          "id": 3182,
          "url": "http://localhost:3000/users/75/edit",
          "title": "Servlet Error"
        },
        {
          "id": 3183,
          "url": "https://docs.racket-lang.org/reference/lambda.html?q=lambda#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._lambda%29%29",
          "title": "3.8 Procedure Expressions: lambda and case-lambda"
        },
        {
          "id": 3185,
          "url": "https://docs.racket-lang.org/reference/exns.html?q=object-name#%28def._%28%28quote._~23~25kernel%29._raise-user-error%29%29",
          "title": "10.2 Exceptions"
        },
        {
          "id": 3188,
          "url": "https://docs.racket-lang.org/reference/ivaraccess.html",
          "title": "6.4 Field and Method Access"
        },
        {
          "id": 3197,
          "url": "http://stackoverflow.com/questions/23296384/application-not-a-procedure-racket",
          "title": "condition - application: not a procedure racket - Stack Overflow"
        }
      ],
      "createDate": 1453743658563
    },
    {
      "id": 3212,
      "tabsMeta": [
        {
          "id": 3199,
          "url": "https://docs.racket-lang.org/reference/exns.html?q=object-name#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._exn~3afail%29%29",
          "title": "10.2 Exceptions"
        },
        {
          "id": 3200,
          "url": "https://docs.racket-lang.org/reference/define-struct.html?q=object-name#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define-struct%29%29",
          "title": "5.1 Defining Structure Types: struct"
        },
        ...
      ],
      "createDate": 1454092984398
    },
     ...
  ]
}

It’s Just a pure Json data. I make a copy of this for backup purposes. Now I can remove some items here. I did few tests to find the recommended size, I prefer to limit to 1000. Besides that make sure the id of the last item is the same as the value of idCounter in itemTable.

Data is ready, But …

You can’t make this patch work if chrome is running. Basically, all the chrome’s instances must be terminated before you are touching the extension data. To do so, you might use some commands like: kill, top or htop, etc.

Finally just replace the state field of itemTable with the modified version of the json data. And of course, OneTab is back!

讀書筆記:閱讀的技法

作者介紹:

佐藤優 ─ 前日本駐俄羅斯外交官,外語除英語外精通捷克語, 韓語及俄羅斯語。初中前讀書量極少,主要看電子類書籍及文學小說。初一後嘗試接觸不同類型書籍,後來在大學研讀神學書和哲學書。現時一個月平圴能讀 300 本,狀態更好能達 500 本。每天需要處理超過 2500 頁 A4 文件,轉換成書的話大概一天要讀完 5 本。

習慣讀書人仕平圴一個月僅可「熟讀」 5 本 ,一年即 60 本,一生才大概 3000 本。由於一生人可以讀的書很有限,因此「速讀」變得非常重要。

速讀守則 :

  1. 必需對該範圍的基礎知識相當穩固,否則只能「熟讀」
  2. 只需要看未懂或新知識的部分
  3. 速讀不是瀏覽

熟讀守則:

  1. 以奇數數目選擇書本,例如 3 本 , 5 本。因為作者間的觀點若果出現衝突,可以採取多數決解決問題,避免進入兩難局面浪費時間。
  2. 試閱書的中間章節,那是書最脆弱的部分。
  3. 集中精力閱讀學術類書籍的最後部分,因為那是作者對該理論總結其證明最有力的部分。
  4. 需要「弄污」書籍做筆記,借用圖書館書籍就不能做,故此建議花點錢買書。
  5. 看書時用鉛筆以方框圈著重要的段落,不宜超過總頁數的 10分1;重要的書頁可以拆角或貼上標籤。
  6. 遇到不明白的地方以 “?” 標注;在書內對作者的觀點寫下個人意見,判斷,一至兩句即可。
  7. 利用方框圈點的重點製作筆記。若果解決了當初不明白的地方,可以把標記擦去。
  8. 同一本書需讀最小 3 次。

讀書術可分成三類:

  1. 超速讀 (5 分鐘),並無需做筆記
  2. 普通速讀 (30 分鐘),做小量筆記
  3. 熟讀 ,做深入筆記