Git sparse-checkout and partial clones for Mega-Repos
2023-05-05

Recently I was planning to make a small contribution to project DefinitelyTyped/DefinitelyTyped. It is a huge repository that hosts and provides the type declaration files for thousands of packages on npm, with a structure like the following

types/
├── 11ty__eleventy-img
├── 1line-aa
├── 3box
├── ...and more of them...

Each sub-folder under the types/ directory corresponds to a specific npm package.

The package I am interested in is marked, whose type declaration files reside in types/marked/ directory. Towards this end, cloning the whole repository and doing a full checkout is considered worthless, since my disk space and network traffic would be eaten up by a bunch of irrelevant files.

Read More

Git-based Dependencies in Dart and Go
2022-07-05

Both Dart and Go support decentralized package distribution. One is able to directly adopt an existing git repository as dependency, easing the effort of distributing packages.

Sometimes we might expect more fine-grained control on what to pull from a git repository. For example, to lock a package’s version, we would specify a particular tag, commit or branch name to pull from. Or if it’s a mono-repo, we would choose a sub-directory from the repository root. This post summarizes how to achieve these purposes in both languages.

Read More

Retrieve Contents over HTTP without curl or wget
2021-10-30

I came across a piece of interesting vulnerable script from a post on V2EX 1) on V2EX. A bash function named __curl inside the file retrieves contents over HTTP as a simple alternative for command curl or wget, in scenarios where no such utilities available.

#!/bin/bash
function __curl() {
  read proto server path <<<$(echo ${1//// })
  DOC=/${path// //}
  HOST=${server//:*}
  PORT=${server//*:}
  [[ x"${HOST}" == x"${PORT}" ]] && PORT=80

  exec 3<>/dev/tcp/${HOST}/$PORT
  echo -en "GET ${DOC} HTTP/1.0\r\nHost: ${HOST}\r\n\r\n" >&3
  (while read line; do
   [[ "$line" == $'\r' ]] && break
  done && cat) <&3
  exec 3>&-
}

The function makes use of certain less known features of Linux and the Bash language.

The first one is communicating over TCP through files. Linux employs a design philosophy of “everything are files”. One could find some special device files in directory /dev, through which we can manipulate the underlying devices. Specifically, manipulating a TCP socket connecting ${HOST}:${PORT} could be achieved by accessing device file /dev/tcp/${HOST}/${PORT}. Since HTTP is a text-based protocol over TCP, working with it is no more difficult than reading / writing a text file. Line exec 3<>$FILENAME opens file $FILENAME under read-write mode and binds it to descriptor 3. The next line then manually composes an HTTP payload and writes out to &3, which is in fact requesting the URL http ://${HOST}:${PORT}. By reading the same file descriptor, we retrieve the response content from the service. The trick serves as a primitive workaround for retrieving contents from web.

Another one is parameter substitution in Bash. The expression ${var//PATTERN/REPL} substitutes all occurrences of PATTERN in var into REPL. If REPL omitted, the matched substrings will be deleted. For example, in this script, ${1//// } would replace all slashes / into white spaces in variable $1.

References

  1. Parameter Substitution
  1. [收到条阿里云的告警，看不懂是做什么用的，请教一下 - V2EX](https://www.v2ex.com/t/811424
Visualizing Correlation
2019-03-10

Say we have a matrix A of shape N x M , which can be viewed as a collection of N vectors of shape 1 x M . The code below gives us the correlation matrix of A :

A_corr = np.corrcoef(A)  # shape: (N, N)

To visualize it, just use plt.matshow(A_corr) .

If N is so large that the figure could not provide a clear insight, we might alternatively use histograms like this:

def corr_matrix_to_array(corr_mat):
    N = corr_mat.shape[0]
    return np.array([corr_mat[i][j] for i in range(1, N) for j in range(i + 1, N)])

plt.hist(corr_matrix_to_array(A_corr), bins=np.linspace(-1, 1, N_bins))
SS Configuration
2019-01-21

SS Client

$ [sudo] pip3 install shadowsocks

/etc/ss.json :

{
    "server": "<server ip>",
    "server_port": "<server port>", // must be Number
    "password": "<password>",
    "local_address": "127.0.0.1",
    "local_port": 1081,
    "timeout": 300,
    "method": "aes-256-cfb",
    "fast_open": false
}
$ [sudo] sslocal -c /etc/ss.json -d start

proxychains

clone repository from https://github.com/rofl0r/proxychains-ng , make && sudo make install .

Append following lines to /etc/proxychains.conf :

[ProxyList]
# add proxy here ...
# meanwile
# defaults set to "tor"
socks5	127.0.0.1	1081

Usage: proxychains [command] .

Chrome Addons

SwitchyOmega .

使用 Aria2 在 Ubuntu 中下载百度云资源
2018-06-11

可以实现满带宽下载。

配置 Aria2

从 Github 下载源码 ./configure && make -j8 && sudo make install。

配置 Chrome 插件

clone https://github.com/acgotaku/BaiduExporter。

$ cd ariac
$ cat > start.sh
> #!/bin/bash
> aria2c --conf=aria2.conf
> ^D
$ chmod +x start.sh
$ ./start.sh

安装 Chrome 插件

打开 chrome://extensions，Load Unpacked 选择 chrome/release。

完成后在百度云页面上会有 导出下载 按钮。

Ubuntu 重新映射键盘布局
2016-06-05

键盘持续失灵，已经到了让我忍无可忍的地步了。

刚开始只是方向键失灵，好在可以用小键盘替代；后来右 Ctrl 和 Alt 也失灵了，好在可以用左边的替代；直到最近 Fn 键也失灵了，终于逼疯了我——因为这意味着 F1 ～ F12 都将不能使用。

我曾试图寻找方法将 CapsLock 键映射为 Fn 键映射，但失败了——Fn 键消息是由 BIOS 拦截的，无法被操作系统捕获。

但今天我找到了一个更好的替代方案：

  • 交换 Fn 和 Ctrl。这是唯一一种能让 Fn 键移位的方式，在所有的 BIOS 中都可以设置。
  • 将 CapsLock 映射为 Ctrl。反正 CapsLock 闲着也是闲着，不如用它代替坏了的键。

ubuntu 下需要执行：

setxkbmap -layout us -option ctrl:nocaps

参考： How do I turn Caps Lock into an extra Control key? - Ask Ubuntu

如何备份 apt-get 已安装的软件列表
2015-04-19

apt-get 是 Ubuntu 下管理软件包的一个工具，实用简单，功能强大。平时若要安装或卸载软件包，只需轻敲一条指令即可。每一台 Ubuntu 上，都安装着数以千百计的软件包——或是内核模块，或是工作、娱乐所需的软件，在它们的支持下，工作着这个开放的操作系统。

但，如果有一天，系统需要被重装——或是无可救药了，抑或是购置了新的设备，问题来了：

如何将现有电脑上的软件包迁移至新的系统呢？

很简单。首先，将原有的软件列表导出：

sudo dpkg --get-selections  > app-backup-list.lst

最好是设置一个定时任务，每隔一段时间就保存一次列表，并且要保存到一个独立的分区。以免某天系统真的坏了。

接下来便是导入了：

sudo dpkg --set-selections < app-backup-list.lst
sudo apt-get -y update
sudo apt-get dselect-upgrade

至于软件源的备份，只需将 /etc/apt/sources.list 文件复制过去即可。