Git sparse-checkout and partial clones for Mega-Repos

Recently I was planning to make a small contribution to project DefinitelyTyped/DefinitelyTyped. It is a huge repository that hosts and provides the type declaration files for thousands of packages on npm, with a structure like the following

types/
├── 11ty__eleventy-img
├── 1line-aa
├── 3box
├── ...and more of them...

Each sub-folder under the types/ directory corresponds to a specific npm package.

The package I am interested in is marked, whose type declaration files reside in types/marked/ directory. Towards this end, cloning the whole repository and doing a full checkout is considered worthless, since my disk space and network traffic would be eaten up by a bunch of irrelevant files.

阅读更多

辩义“封建”

“封建”一词很有意思。作为一个土生土长的中国词,它于千年前便有了固定而确切的含义。然而在近代西方思潮的冲击下,这个词的意义先是遭到了替换,而后又在近百年的传播中演化成了与本义毫不相干的概念。如今的人们谈及封建一词时,心中不免会有困惑,因为其现代含义与汉字构成相去甚远。更糟糕的是,在一些语境下,这个词会回归至其近代甚至是古代的含义,从而对相关的交流造成混乱。同一词的释义在多种语境下不一致,无谓的争吵便在所难免。

粗略地说,“封建”至少有四种释义,即古代中国的封建制度、古代欧洲的 Feudalism、近现代中国与 Feudalism 的对译,以及当代民间语境中的封建。

阅读更多

Diving from the CUDA Error 804 into a bug of libnvidia-container

Several users reported to encounter "Error 804: forward compatibility was attempted on non supported HW" during the usage of some customized PyTorch docker images on our GPU cluster.

At first glance I recognized the culprit to be a version mismatch between installed driver on the host and required driver in the image. The corrupted images as they described were built targeting CUDA == 11.3 with a corresponding driver version == 465 , while some of our hosts are shipped with driver version 460. As a solution I told them to downgrade the targeting CUDA version by choosing a base image such as nvidia/cuda:11.2.0-devel-ubuntu18.04, which indeed well solved the problem.

But later on I suspected the above hypothesis being the real cause. An observed counterexample was that another line of docker images targeting even higher CUDA version would run normally on those hosts, for example, the latest ghcr.io/pytorch/pytorch:2.0.0-devel built for CUDA == 11.7. This won’t be the case if CUDA version mismatch truly matters.

Afterwards I did a bit of research concerning the problem and learnt some interesting stuff which this post is going to share. In short, the recently released minor version compatibility allows applications built for newer CUDA to run on machines with some older drivers, but libnvidia-container doesn’t correcly handle it due to a bug and eventually leads to such an error.

Towards thorough comprehension, this post will first introduce the constitution of CUDA components, following with the compatibility policy of different components, and finally unravel the bug and devise a workaround for it. But before diving deep, I’ll give two Dockerfile samples to illustrate the problem.

阅读更多

Modern Cryptography, GPG and Integration with Git(hub)

GPG (the GNU Privacy Guard) is a complete and free implementation of the OpenPGP standard. Based on various mature algorithms to select from, GPG acts as a convenient tool for daily cryptographic communication.

GPG has two primary functionalities: (1) it encrypts and signs your data for secure transfering and verifiable information integrity, and (2) it features a versatile key management system to construct and promote web of trust. GPG also has a well-designed command line interface for easy integration with other applications such as git.

This article is going to briefly elaborate some key concepts and usage of GPG, and then present demonstration to cryptographically sign git commits with the help of GPG.

阅读更多

Move the Root Partition of Ubuntu

Some days ago, I made the decision to shrink the footprint of Windows system on my laptop and reallocate the disk space to the Ubuntu system that resides next to it. Ubuntu is competent for my daily use of programming and web browsing so I hardly launched the OEM-shipped Windows since the laptop was bought. The Windows takes up a not-so-small portion of my SSD space, which can be better utilized instead of wasted in vain.

#gk:immersive
(before)
| --- Windows C: (256 GB) --- | --- Ubuntu / (256 GB) --- |
(after)
| --- Windows C: (120 GB) --- | --- Ubuntu / (392 GB) --- |
阅读更多

A New Programmer Kicks a Roadblock

The time I composed my first program can be back to my junior high school age. It was the first day of PC lesson, and everybody crowded to the computer classroom. We were told to learn “programming” there. The kids who were talented would be selected and trained for OI . Others instead would go to an ordinary class and learn something more general.

I was anxious. Before the time I had no concept of what “programming” is, nor had I ever gone through a real PC lesson. The PC lesson in my primary school barely taught anything. Over the time the teachers let us play games instead. I could type merely a dozen of characters per minute, since I’d never received a thorough typing training. I was ignorant of inside the metal box. I was a complete computer idiot.

阅读更多

Git-based Dependencies in Dart and Go

Both Dart and Go support decentralized package distribution. One is able to directly adopt an existing git repository as dependency, easing the effort of distributing packages.

Sometimes we might expect more fine-grained control on what to pull from a git repository. For example, to lock a package’s version, we would specify a particular tag, commit or branch name to pull from. Or if it’s a mono-repo, we would choose a sub-directory from the repository root. This post summarizes how to achieve these purposes in both languages.

阅读更多

Reversy Naming

I am always a dedicated fan of writing naturally readable code – by “naturally readable” I mean, one can read a line of code as if it were a sentence of English (or maybe other human languages). It’s believed that the practice encourages more self-explainable code, as the code reads more like a human-composed article, instead of some gibberish only recognizable by machine.

The practice recommends to name functions or variables following the word order of human language, for English that is, subjects come after verbs, and adjectives go before nouns that being modified. The samples below showcase how it guides naming in a program (please hold your opinions about the casing)

  • append_to_list(lst, item). A function that appends an item to a list, which can read as “append to the list (specified by name lst) with the item”.
  • register_service_notifier(func). A function that registers another function as a service notifier, which can read as “register a service notifier with the function func“.
  • UserFollowersListView. The name of a web component which is a list view to display followers for a user.

It plays well and improves my developing experience most of the time, but there is no silver bullet, just like other practices or guidelines.Sometimes I found the readability even degrades. I kept skimming the lines and just couldn’t locate an item efficiently.

阅读更多

人类一败涂地

轻微的一阵异响。面前说话的她随即模糊起来,声音也渐渐远去,趋于缥缈。我这才注意到从一开始就没有看清她的脸——原来只是个梦罢了。意识跌落回现实。我又重新感受到了我的双腿,久坐后腰的酸痛,最后是沉重的眼皮。睁开眼,还是熟悉的银色舷窗。

窗外的天王星已经很大了。坐了这么多次,光看天王星的大小也能知道到哪了,估计还得两个小时才能下船。我伸了个懒腰,腿上的便携式电脑悠了起来,屏幕被点亮了。光标停在了讲稿的最后一行,冷冷地跳动着。

再读一遍吧。到那边可有得我忙的。

周围的人大多和我一样,要么在闭目养神,要么在低头对着电脑,大家都一言不发。长途旅行最能磨去一个人的精气,但为了工作我不得不奔波于各个星球之间——真是个乏味的时代。但有一个人不一样。

“孩子,那就是天王星了吧?”

阅读更多

Invalid Golang Pointers Can Bite You Even If You Don't Dereference

In Golang, if you coerce a uintptr variable into unsafe.Pointer (or further, to some *T), the linter will warn with the message "possible miuse of unsafe.Pointer". This makes sense because the uintptr variable may contain an address that points to a piece of invalid memory, and dereferencing such a pointer is catastrophic (usually aborts the program).

I was always aware of the above discipline, but I thought it would be OK to hold the pointers but not dereference them. This is true in C/C++, but not for Golang, which I did not realize until recently.

In fact, the program can panic even if you just keep an invalid pointer on the stack!

阅读更多