Blog

LLM can accumulate cognitive debt

Wed, 01 Oct 2025 00:00:00 +0200

I read “Your Brain on ChatGPT: Accumulation of Cognitive Debt When Using an AI Assistant for Essay Writing Task.” ¹ It’s useful for deciding when to use LLMs in research and writing.

Cognitive debt is when repeated reliance on external systems (like LLMs) displaces the effortful processes of independent thinking - saving effort now but costing depth, recall, and agency later.

The study finds higher neural engagement when drafting unaided. So think first, write first, then ask an LLM for critique, revise, or fact-check.

What the study did

Participants completed three essay-writing sessions in one of three conditions, plus an crossover session. Essays were written under electroencephalogram (EEG), with NLP and interviews used for analysis. N=54 completed sessions 1-3; N=18 returned for session 4. Each essay takes about 20 minutes. The 3 groups were:

LLM: GPT-4o only
Search Engine: Google with “-ai” to suppress AI answers
Brain-only: no tools

The study ran over 4 months.

What they found

I am interested in how should we use the technology itself from reading the paper. So I will only choose relevant conclusions. The paper offers a result of comparisons.

First of all, there is evidence indicating that the brain-only group write essays using their brain “harder”.

For example, brain-only participants showed higher activation levels in EEG and greater overall cognitive load. Writing unaided requires the brain to generate ideas, organize, and compose ideas internally. By contrast, with LLM assistance - though the essay often scored higher and were easier to write - the LLM group exerted less effort in self-monitoring. They also showed weaker memory traces and fragmented authorship.

Moreover, participants in the Brain-to-LLM group (writing first unaided, then revising with LLM) showed greater meta-cognitive engagement. They likely invested more effort in coordinating with the model and making edits to LLM’s output.

Finally, the paper finds a link between neural connectivity and behavioral quoting failures. Participants more reliant on tools encoded information more shallowly. However, meta-cognitive engagement was generally higher for the Brain-to-LLM group.

How to use LLM without “cognitive debt”

In a word, when writing something new, it’s better to draft firstly, unaided and ask an LLM to check or refine your work.

For “vibe coding” or day-to-day drafting, letting an agent like Claude run might not be the best approach for writing and learning.

Using your brain earlier helps you memorize and think deeper. Using agents or LLM engages us differently. Still, quality can be significantly improved by layering in LLM checks at the end.

Kosmyna, Nataliya, Eugene Hauptmann, Ye Tong Yuan, et al. “Your Brain on ChatGPT: Accumulation of Cognitive Debt When Using an AI Assistant for Essay Writing Task.” arXiv:2506.08872. Preprint, June 10, 2025. https://doi.org/10.48550/arXiv.2506.08872. ↩

Latency values for programmers in 2024

Thu, 26 Sep 2024 00:00:00 +0200

Latency matters in software. As hardware evolves, the assumptions we make about computer systems must evolve with it. Programs often spend time waiting—on disk or network I/O, on another thread to finish work, or on a remote service to respond. Within a system, a database lock can stall other connections, and a global interpreter lock (GIL) can block parallel threads. These forms of contention are common and important to address. Although hardware has grown dramatically faster over the decades, the fundamental latency challenges remain. Below is what I learned in 2024.

Operation	Time in nanosecond	Time in microsecond	Time in millisecond	Comparison	Reference
L1 cache reference	0.7				Zen 5 numbers
L2 cache reference	2.5
Branch mispredict	8
L3 cache reference	8
Mutex lock/unlock	25
Main memory reference	70			100x L1
Send 2 kB over 10 Gbps network	1600	1.6
Compress 1K bytes with Snappy	2000	2
Read 1 MB sequentially from memory	10000	10			~50 GB/s DDR5
Read 4K randomly from SSD	20000	20			~10 GB/s NVMe
Read 1 MB sequentially from NVMe	100000	100
Round trip within same datacenter	100000	100			~0.5GB/sec SSD, 100x memory, 20x NVMe
SSD seek time	160,000	160		2,000x main memory	~150MB/sec
Read 1 MB sequentially from SSD	2,000,000	2000	2
AWS region round trip	2,000,000	2000	2
Read 1 MB sequentially from HDD	6,000,000	6000	6
Send 1 MB over 1 Gbps network	10,000,000	10000	10
HDD random seek time	10,000,000	10000	10		Often times, data are located close by. The typical seek time should be lower.
Send packet CA->Netherlands->CA	150,000,000	150000	150

Storage device and compute device innovations are most relevant to software applications. Apart from latency numbers, DDR5 runs at 6,000 MT/s, so this is 48,000 MB/s (random access). SSDs perform surprisingly well in terms of sequential read/write nowadays.

Sources
https://static.googleusercontent.com/media/sre.google/en//static/pdf/rule-of-thumb-latency-numbers-letter.pdf
https://gist.github.com/BlackHC/2d0a3a21542b524a7cf2f8eac977481e
https://colin-scott.github.io/personal_website/research/interactive_latency.html
https://cloud.ibm.com/docs/vpc?topic=vpc-network-latency-dashboard
https://en.wikipedia.org/wiki/Hard_disk_drive_performance_characteristics

WebAssembly vs. JVM: An Informal Peek

Thu, 18 May 2023 00:00:00 +0200

Recently, WebAssembly (WASM) has gained traction as a reference runtime, providing a way to run programs written in different programming languages. However, the Java Virtual Machine (JVM) is another popular platform for achieving similar functionality. When it comes to running programs on a virtual machine or compiling them into machine code, it’s also worth considering LLVM. In this blog post, I’ll share some thoughts based on my reading, although it’s not a complete analysis but rather a collection of ideas to encourage thinking and exploration.

Representation

LLVM offers three different forms: an in-memory compiler Intermediate Representation (IR), an on-disk bitcode representation (optimized for fast loading by Just-In-Time compilers), and a human-readable assembly language representation. It’s important to note that LLVM itself is not a runtime. On the other hand, the JVM uses a binary representation for classes and interfaces, while WASM utilizes binary and text formats.

Despite the differences in representation, the core principle of linking a program and executing it remains the same. Instructions and data can be represented in either text or binary formats on disk. The program must be loaded into memory, either dynamically or statically, before execution can take place.

In my exploration, I found that the WASM specification surprisingly offers a simpler approach compared to LLVM and JVM. This simplicity may stem from the focus on optimization and features within the LLVM and JVM ecosystems. It’s worth noting that WASM is relatively new and primarily targets browser and integrated environments. Additionally, WASM’s streamability sets it apart from other tools.

Modules in WASM

WASM organizes code structure into modules, which serve as the unit of loading and compilation. Unlike JVM, modules are not equivalent to interfaces or classes. Instead, a module encompasses types, functions, memory, global variables, data, and execution symbols. In contrast, JVM represents these structures through a ClassFile, which contains metadata such as types, constants, methods, fields, and attributes. Instructions are stored as an attribute. LLVM’s representation, on the other hand, aligns more closely with machine code. Ultimately, LLVM supports compiler backends to translate its representation into machine code.

Instructions in WASM

Based on the high-level constructs mentioned earlier, WASM offers various types of instructions, including numerical, memory, table, variable, parametric, reference, vector, and control instructions. Some of these instructions resemble assembly code, while others bear similarities to JVM instructions. However, JVM includes additional instructions for handling null values, basic types, exceptions, and inheritance-related operations.

By comparing these different platforms and their instruction sets, we can gain a better understanding of their respective strengths and use cases. How it is built is in details of course.

Implementation

wasmtime serves as a standalone runtime for WebAssembly (WASM) and provides a clear structure outlined in its Architecture document. The runtime revolves around the concept of an Engine, which serves as the global compilation context. The Store maintains the context and isolates WASM objects within a thread. The document also explains the compilation and execution processes, although the validation process will be excluded from this blog.

Similar to the JVM, the runtime keeps track of all loaded modules. In the JVM, code is typically compiled into .class files, and the JVM and class loaders collaborate to load symbols and instructions for a program. On the other hand, wasmtime seems to have a module linking procedure. While the multithreaded environment of the JVM is often discussed, the details regarding multithreading in wasmtime are not mentioned as prominently, which piques my interest.

Discussions

WASM defines a straightforward structure that allows programs to run on a runtime (or even without one). This approach facilitates the creation of desired characteristics across multiple programming languages. It differs from the approach taken by GraalVM. The ability to run different programming languages within a single runtime can significantly increase productivity. Integrating various programs and libraries into one runtime fosters better integration and reduces the cost of building blocks.

Popular connectors for computers

Sun, 28 Jun 2020 00:00:00 +0200

^{Pin out tables, some photos and parts of data come from Wikipedia.}

Computers have various of connectors for their diverse use case. Overtime, old standard become obselete and new connector emerges. Some of them established their ground and dominates in their domains. It’s interesting to note their capability and how they got where they are. I am going to list some of them in this post and summarize key features. If you know a little bit about past standards, you might be amazed how technology eveolves step by step. Had if you know nothing about past standards, these connectors and standards would become old in the next decade or further in the future.

USB

Starting from a golden standard of physical peripheral, this connector is Universal Serial Bus (USB). USB is everywhere. You can find them in laptops, cameras, drones, headsets, gaming consoles, and even industry machines. For example, latest Macbook Pro (2020) has 4 USB-C while their old predecessors have Thunderbolt connectors. Nowadays, most Android phones come with a USB-C connector.

USB has become complicated and confusing industry standard. The confusing part comes with multiple physical connector types and its different versions. Motherboards usually have different types of USB 3.x receptacles or jump wires. Despite its historical burden, the latest version, USB-C aims to move forward with all advantages and drawbacks from its past versions and receptacles. In the past, USB’s standard series are widely used. It has a larger connector than other series which is called Type-A. Type-A usually presents on a PC. Type-B has a more or less square shape. It is usually used in a display. Apart from those large receptacles, the mini series can be generally found in mobile phones. You can find a table on Wikipedia about different versions and receptacles. The interesting part of the chart is USB-C and USB4 as they are the only combination that are not deprecated after 2019.

USB offers comfort to users. Devices can be hot-plugged. Devices can configure themselves with operating systems and take power from USB cables. USB also provides great abstraction for manufacturers and developers. They can focus on device functions instead of physical connectors and protocol designs.

USB4 is the current standard which has only 1 receptacle (Type-C) with many technologies packed together. Considering USB 3.x series, USB 3.0, 3.1, 3.2 specification support a transfer mode of 5, 10, 20 Gbit/s. By the way, they were rebranded with a new set of confusing names, USB 3.2 Gen 1, USB 3.2 Gen 2, USB 3.2 Gen 2x2. Speaking of USB’s simplicity, the user shall be able to connect to a receptacle and expect a device to set up itself. You are not suppose to connect a cable to a USB recepatacle and wonder why things were wrong. With past standards, you wouldn’t be able to connect a cable that does not fit the receptacles. But their speed difference can be huge. For instance, USB disks can be extremly slow with USB 2 receptacle and USB 3 cable. That won’t be the problem for USB 4 (Type-C).

Type-C has 24 pins other than USB3’s 9 pins. Physically, USB Type-C receptacle looks smaller than its predecessor. It’s thin and packed.

USB-C receptacle A pin layout
Pin	Name	Description
A1	GND	Ground return
A2	SSTXp1	SuperSpeed differential pair #1, TX, positive
A3	SSTXn1	SuperSpeed differential pair #1, TX, negative
A4	V_BUS	Bus power
A5	CC1	Configuration channel
A6	Dp1	USB 2.0 differential pair, position 1, positive
A7	Dn1	USB 2.0 differential pair, position 1, negative
A8	SBU1	Sideband use (SBU)
A9	V_BUS	Bus power
A10	SSRXn2	SuperSpeed differential pair #4, RX, negative
A11	SSRXp2	SuperSpeed differential pair #4, RX, positive
A12	GND	Ground return

USB-C receptacle B pin layout
Pin	Name	Description
B12	GND	Ground return
B11	SSRXp1	SuperSpeed differential pair #2, RX, positive
B10	SSRXn1	SuperSpeed differential pair #2, RX, negative
B9	V_BUS	Bus power
B8	SBU2	Sideband use (SBU)
B7	Dn2	USB 2.0 differential pair, position 2, negative</sup>
B6	Dp2	USB 2.0 differential pair, position 2, positive</sup>
B5	CC2	Configuration channel
B4	V_BUS	Bus power
B3	SSTXn2	SuperSpeed differential pair #3, TX, negative
B2	SSTXp2	SuperSpeed differential pair #3, TX, positive
B1	GND	Ground return

Type-C has 4 power and 4 ground pins, 2 differential pairs for USB 2.0 data, 4 shielded differential pairs for USB 3.x data, and other pins. With USB Power Deliviery standard, a series of power profiles are supported. So a USB cable and power source can support 0.5W to 100W. Of course, it depends on the cable, host and devices for the power and data speed. USB 4 shall provide a maximum 40Gbit data link.

USB 4 introduces 5 alternate modes to support converters and other protocols. There are 3 types of USB Type-C cables. Firstly, full featured Type-C cable supports USB3.2 performance and other alternate connection such as DisplayPort, MHL, HDMI and Thunderbolt. Secondly, Thunderbolt 3 Type-C to Type-C (40Gbit/s) is another type of cable which can only be used as USB2 cable other than Thunderbolt 3. Lastly, adapter cables will be available. They will have their signs indicating their ability. The best option seems to be use full featured USB 4 Type-C cables and connectors. Otherwise, we have to respect what type of USB cables are. They might not work as expected even they look like the same.

Lightning

Apple built Lightning connector and used it in every Apple mobile devices since 2012. It’s a great connector for its use. Lightning connector is not even replaced by USB-C while Macbook Pro strives to push forward the industry standard practice.

Lightning Connector pin outs
Pin 1	GND	Ground
Pin 2	L0p	Lane 0 positive
Pin 3	L0n	Lane 0 negative
Pin 4	ID0	Identification/control 0
Pin 5	PWR	Power (charger or battery)
Pin 6	L1n	Lane 1 negative
Pin 7	L1p	Lane 1 positive
Pin 8	ID1	Identification/control 1
Lane 0 and 1 may swap in IC of device connector (lanes don't swap if the accessory identification chip is connected to the ID0 pin)

While the connector has certain layout, Lightning cables can be adapative because of Apple certified chip included in the cable. All pins seem to be able to switch between data and power. iPhone supports fast charging so the connector can support 18W charger at least. However, the speed is a secret. Sidecar feature which extends macOS screen to an iPad shows the eviendence of a fast transfering speed.

HDMI

HDMI is usually used to connect a display device which is surprisingly popular. It’s not replaced by thunderbolt/USB4 yet, allegedly because of its price advantage. Being a audio/video transmission interface, HDMI has its roots with its predecessor DVI. Starting from HDMI 2.1, its encoding becomes similiar to USB3 and supports 48Gbit/s data transfering speed.

HDMI (High-Definition Multimedia Interface) Type A Receptacle Pin out
Pin 1	TMDS Data2+
Pin 2	TMDS Data2 Shield
Pin 3	TMDS Data2−
Pin 4	TMDS Data1+
Pin 5	TMDS Data1 Shield
Pin 6	TMDS Data1−
Pin 7	TMDS Data0+
Pin 8	TMDS Data0 Shield
Pin 9	TMDS Data0−
Pin 10	TMDS Clock+
Pin 11	TMDS Clock Shield
Pin 12	TMDS Clock−
Pin 13	CEC
Pin 14	Reserved (HDMI 1.0–1.3a) Utility/HEAC+ (HDMI 1.4+, optional, HDMI Ethernet Channel and Audio Return Channel)
Pin 15	SCL (I²C serial clock for DDC)
Pin 16	SDA (I²C serial data for DDC)
Pin 17	Ground (for DDC, CEC, ARC, and HEC)
Pin 18	+5 V (min. 0.055 A)
Pin 19	Hot Plug Detect (all versions) HEAC− (HDMI 1.4+, optional, HDMI Ethernet Channel and Audio Return Channel)

SATA

SATA (Serial AT Attachment) can be commonly found with storage devices. Various of hard drives and SSDs use SATA for power and data transfer. M.2 connector may also use SATA bus but those devices does not use SATA connectors.

Its data connector has 7 pins and is much smaller than PATA connector.

SATA Standard connector, data segment
Pin #		Mating	Function
	1	1st	Ground
	2	2nd	A+ (transmit)
	3	2nd	A− (transmit)
	4	1st	Ground
	5	2nd	B− (receive)
	6	2nd	B+ (receive)
	7	1st	Ground
—			Coding notch

Its power connector has a 15 pin connector with various of voltages.

SATA Standard connector, power segment
Pin #		Mating	Function
—			Coding notch
	1	3rd	3.3 V Power
	2	3rd	3.3 V Power
	3	2nd	Enter/exit Power Disable (PWDIS) mode (3.3 V Power, Pre-charge prior to SATA 3.3)
	4	1st	Ground
	5	2nd
	6	2nd
	7	2nd	5 V Power, Pre-charge
	8	3rd	5 V Power
	9	3rd	5 V Power
	10	2nd	Ground
	11	3rd	Staggered spinup/activity
	12	1st	Ground
	13	2nd	12 V Power, Pre-charge
	14	3rd	12 V Power
	15	3rd	12 V Power

The latest standard supports 6Gbit/s transfering speed.

PCI Express

PCI-E has been the primary choices for devices that are inter-connected on the motherboard. PCI-E device communicates point-to-point with each other and use packets. PCI-E link varies from one to 32 lanes. Devices can be plugged into another PCI-E sockets. Link speed is negotiated to the maximum possible speed.

PCI Express connector pinout (x1, x4, x8 and x16 variants)
Pin	Side B	Side A	Description	Pin	Side B	Side A	Description
01	+12 V	PRSNT1#	Must connect to farthest PRSNT2# pin	50	HSOp(8)	Reserved	Lane 8 transmit data, + and −
02	+12 V	+12 V	Main power pins	51	HSOn(8)	Ground	Lane 8 transmit data, + and −
03	+12 V	+12 V	Main power pins	52	Ground	HSIp(8)	Lane 8 receive data, + and −
04	Ground	Ground		53	Ground	HSIn(8)	Lane 8 receive data, + and −
05	SMCLK	TCK	SMBus and JTAG port pins	54	HSOp(9)	Ground	Lane 9 transmit data, + and −
06	SMDAT	TDI		55	HSOn(9)	Ground	Lane 9 transmit data, + and −
07	Ground	TDO		56	Ground	HSIp(9)	Lane 9 receive data, + and −
08	+3.3 V	TMS		57	Ground	HSIn(9)	Lane 9 receive data, + and −
09	TRST#	+3.3 V		58	HSOp(10)	Ground	Lane 10 transmit data, + and −
10	+3.3 V aux	+3.3 V	Standby power	59	HSOn(10)	Ground	Lane 10 transmit data, + and −
11	WAKE#	PERST#	Link reactivation; fundamental reset	60	Ground	HSIp(10)	Lane 10 receive data, + and −
Key notch				61	Ground	HSIn(10)	Lane 10 receive data, + and −
12	CLKREQ#	Ground	Clock Request Signal	62	HSOp(11)	Ground	Lane 11 transmit data, + and −
13	Ground	REFCLK+	Reference clock differential pair	63	HSOn(11)	Ground	Lane 11 transmit data, + and −
14	HSOp(0)	REFCLK−	Lane 0 transmit data, + and −	64	Ground	HSIp(11)	Lane 11 receive data, + and −
15	HSOn(0)	Ground	Lane 0 transmit data, + and −	65	Ground	HSIn(11)	Lane 11 receive data, + and −
16	Ground	HSIp(0)	Lane 0 receive data, + and −	66	HSOp(12)	Ground	Lane 12 transmit data, + and −
17	PRSNT2#	HSIn(0)	Lane 0 receive data, + and −	67	HSOn(12)	Ground	Lane 12 transmit data, + and −
18	Ground	Ground		68	Ground	HSIp(12)	Lane 12 receive data, + and −
PCI Express x1 cards end at pin 18				69	Ground	HSIn(12)	Lane 12 receive data, + and −
19	HSOp(1)	Reserved	Lane 1 transmit data, + and −	70	HSOp(13)	Ground	Lane 13 transmit data, + and −
20	HSOn(1)	Ground	Lane 1 transmit data, + and −	71	HSOn(13)	Ground	Lane 13 transmit data, + and −
21	Ground	HSIp(1)	Lane 1 receive data, + and −	72	Ground	HSIp(13)	Lane 13 receive data, + and −
22	Ground	HSIn(1)	Lane 1 receive data, + and −	73	Ground	HSIn(13)	Lane 13 receive data, + and −
23	HSOp(2)	Ground	Lane 2 transmit data, + and −	74	HSOp(14)	Ground	Lane 14 transmit data, + and −
24	HSOn(2)	Ground	Lane 2 transmit data, + and −	75	HSOn(14)	Ground	Lane 14 transmit data, + and −
25	Ground	HSIp(2)	Lane 2 receive data, + and −	76	Ground	HSIp(14)	Lane 14 receive data, + and −
26	Ground	HSIn(2)	Lane 2 receive data, + and −	77	Ground	HSIn(14)	Lane 14 receive data, + and −
27	HSOp(3)	Ground	Lane 3 transmit data, + and −	78	HSOp(15)	Ground	Lane 15 transmit data, + and −
28	HSOn(3)	Ground	Lane 3 transmit data, + and −	79	HSOn(15)	Ground	Lane 15 transmit data, + and −
29	Ground	HSIp(3)	Lane 3 receive data, + and −	80	Ground	HSIp(15)	Lane 15 receive data, + and −
30	PWRBRK#	HSIn(3)	Lane 3 receive data, + and −	81	PRSNT2#	HSIn(15)	Lane 15 receive data, + and −
31	PRSNT2#	Ground		82	Reserved	Ground
32	Ground	Reserved
PCI Express x4 cards end at pin 32
33	HSOp(4)	Reserved	Lane 4 transmit data, + and −
34	HSOn(4)	Ground	Lane 4 transmit data, + and −
35	Ground	HSIp(4)	Lane 4 receive data, + and −
36	Ground	HSIn(4)	Lane 4 receive data, + and −
37	HSOp(5)	Ground	Lane 5 transmit data, + and −
38	HSOn(5)	Ground	Lane 5 transmit data, + and −
39	Ground	HSIp(5)	Lane 5 receive data, + and −
40	Ground	HSIn(5)	Lane 5 receive data, + and −
41	HSOp(6)	Ground	Lane 6 transmit data, + and −
42	HSOn(6)	Ground	Lane 6 transmit data, + and −
43	Ground	HSIp(6)	Lane 6 receive data, + and −	Legend
44	Ground	HSIn(6)	Lane 6 receive data, + and −	Ground pin		Zero volt reference
45	HSOp(7)	Ground	Lane 7 transmit data, + and −	Power pin		Supplies power to the PCIe card
46	HSOn(7)	Ground	Lane 7 transmit data, + and −	Card-to-host pin		Signal from the card to the motherboard
47	Ground	HSIp(7)	Lane 7 receive data, + and −	Host-to-card pin		Signal from the motherboard to the card
48	PRSNT2#	HSIn(7)	Lane 7 receive data, + and −	Open drain		May be pulled low or sensed by multiple cards
49	Ground	Ground		Sense pin		Tied together on card
PCI Express x8 cards end at pin 49				Reserved		Not presently used, do not connect

Summary

Connectors have different shapes optimized for different tasks. When devices or system evovles, connectors follows. A standard might try to extend its usage with designs but it does not fit every devices or future use cases. All these connectors provide essential functions such as power delivery, data transfering, some sorts of confugration pins. The way they were designed depends on its historical design and constraints.

A decade with Wikipedia

Tue, 14 May 2019 00:00:00 +0200

I’ve spent a decade with Wikipedia.

I was lucky to know internet when I was young. And it didn’t take me long to know Wikipedia. When I started my journey with Wikipedia at 2008, it was quite new for Chinese too. Growing with Chinese Wikipedia community certainly gave me a very different value than my peer classmates. I just didn’t know we were growing with an new era in Chinese history.

Chinese Wikipedia has dramatic “modern” value comparing to the government. We adopt Five pillars. However, seeking consesus is not a trivial task for the community.

History

Chinese Wikipedia was established at 2002. Few people chose the translation for “wiki”(维基). It gets blocked next year. What kind of world they lived in? P.R.China ranks the sixth on GDP. iPhone doesn’t exist.

At that time, surfing on the internet was expensive and difficult. It was probabaly the elites can access the internet. Key people in the community laid out the foundation and governace model for Chinese Wikipedia. Over time, idealists were replaced by rational people following every bits of protocol. Once people started auditing every change, new people would feel attacked by this culture for being unfamiliar with the rules. With ever growing power in the community, the change on governance stagnate. English Wikipedia has more powerfule mechanics like Arbitration Committee. Smaller Wikipedia might have knowledgeable and kind volunteers (e.g. Swedish Wikipedia).

Despite the downside, Chinese Wikipedia grows stronger. The significant engineering effort brought two major language varaints together. That marked history. People from Taiwan, Hong Kong, and Singapore can contribute to Wikipedia together. And rules among political conflicts were established.

Access problem

But in the reality, most of contributors are students who has abduant free time. Nevertheless, some students contributed professional articles but they were few. It’s unlikely for us to get more professional article in science. When the community became more picky, it created higher barrier for writing. In contray, students can only write about traffic, and ACG (Anime, Comics, and Games).

When the government finally started to block Chinese Wikipedia, it’s a working community consist of people from different background and views.

A darker image

Appearntly, Wikipedia communities in Asia don’t grow much as communities in western countries. There might be some reasons. People have less time because of economy. There are many conflicts in society. Educated people are also limited. Moreover, in particular of Chinese Wikipedia, political influence from government is strong. Despite direct influence on Wikipedia, Baidu launched their own encylopedia that has 16 million articles now (as of 2019). So the audience even become less aware about the quality on references.

Where we are at?

Wikipedia communities have become strong. Successful conferences were hold. Wikimedia affliates have large impact. But the technology is lacking. Running such large scale of project, only a few people are working on strategy of tracks.

How PyTorch implements DataParallel?

Tue, 05 Mar 2019 00:00:00 +0100

PyTorch can send batches and models to different GPUs automatically with DataParallel(model). How is it possible? I assume you know PyTorch uses dynamic computational graph as well as Python GIL. And PyTorch version is v1.0.1.

This is a complicated question and I asked on the PyTorch forum. I got a reply from Sebastian Raschka.

TL;DR:

PyTorch trys hard in zero-copying. DataParallel splits tensor by its total size instead of along any axis.

`DataParallel` interface

Module defines its constructor and forward function. And DataParallel does the same. Let’s focus on forward.

def forward(self, *inputs, **kwargs):
    # ...
    inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids) # Step 1, 2
    # ...
    self.module(*inputs[0], **kwargs[0]) # Build graph
    replicas = self.replicate(self.module, self.device_ids[:len(inputs)]) # Step 3
    outputs = self.parallel_apply(replicas, inputs, kwargs) # Step 4
    return self.gather(outputs, self.output_device) # Step 5

There are 4 methods extracted and defined as instance methods which are scatter, replicate, gather and parallel_apply. They executes the step 1, 2; step 3; step 4; step 5 respectively.

Firstly, we have to recognize PyTorch utilize dynamic computational graph. That’s why module is invoked to build nodes. It builds the graph, parameters, grads and buffers.

And how `Function` works

Before we start to look into diffierent steps, let’s look at Function. We are going to see it many times.

class Function(with_metaclass(FunctionMeta, _C._FunctionBase, _ContextMethodMixin, _HookMixin)):
    @staticmethod
    def forward(ctx, *args, **kwargs):
        r"""...
        It must accept a context ctx as the first argument, followed by any
        number of arguments (tensors or other types).

        The context can be used to store tensors that can be then retrieved
        during the backward pass.
        """
        raise NotImplementedError

    @staticmethod
    def backward(ctx, *grad_outputs):
        raise NotImplementedError

Some helper functions used Function as the base class. Moreover, since you will see Function.apply a lot in the following, we should see how it works. (metaclass is tricky.)

def with_metaclass(meta, *bases):
    """Create a base class with a metaclass."""
    # This requires a bit of explanation: the basic idea is to make a dummy
    # metaclass for one level of class instantiation that replaces itself with
    # the actual metaclass.
    class metaclass(meta):
        def __new__(cls, name, this_bases, d):
            return meta(name, bases, d)
    return type.__new__(metaclass, 'temporary_class', (), {})

This is tricky. We have to start from Python’s object design. Starting from Python 3, everything is an object. And all objects are constructed by class type(name, bases, dict). Each parameter would become class object’s __name__, __bases__, and __dict__. So type is the dynamic way to build a class. with_metaclass is a helper to build a metaclass without defining __dict__ yet. Namely, with_metaclass(FunctionMeta, _C._FunctionBase, _ContextMethodMixin, _HookMixin)) reads FunctionMeta is the metaclass but the bases are in the tuple.

Then we are not far away from how it works.

class FunctionMeta(type):
    def __init__(cls, name, bases, attrs):
        // ...
        backward_fn = type(name + 'Backward', (BackwardCFunction,), {'_forward_cls': cls})
        setattr(cls, '_backward_cls', backward_fn)

        return super(FunctionMeta, cls).__init__(name, bases, attrs)

Basically a backward_fn is defined and stored in _backward_cls. Its base class is simple.

class BackwardCFunction(_C._FunctionBase, _ContextMethodMixin, _HookMixin):
    # ...
    def apply(self, *args):
        return self._forward_cls.backward(self, *args)

It will invoke the Function class’s backward method. Therefore _C._FunctionBase is more interesting. C._FunctionBase is defined by THPFunctionType. And the THPFunction defines its data structure.

static struct PyMethodDef THPFunction_methods[] = {
  {(char*)"apply", (PyCFunction)THPFunction_apply, METH_CLASS | METH_VARARGS, nullptr},
  {(char*)"_do_forward", (PyCFunction)THPFunction_do_forward, METH_VARARGS, nullptr},
  {(char*)"_do_backward", (PyCFunction)THPFunction_do_backward, METH_VARARGS, nullptr},
  {(char*)"_register_hook_dict", (PyCFunction)THPFunction__register_hook_dict, METH_O, nullptr},
  {(char*)"register_hook", (PyCFunction)THPFunction_register_hook, METH_O, nullptr},
  {nullptr}
};

It defines the apply method. That’s something we are interested.

PyObject *THPFunction_apply(PyObject *cls, PyObject *inputs)
{
  // ...
  THPObjectPtr backward_cls(PyObject_GetAttrString(cls, "_backward_cls"));
  if (!backward_cls) return nullptr;
  THPObjectPtr ctx_obj(PyObject_CallFunctionObjArgs(backward_cls, nullptr));
  if (!ctx_obj) return nullptr;
  THPFunction* ctx = (THPFunction*)ctx_obj.get();

  // Prepare inputs and allocate context (grad fn)
  auto info_pair = unpack_input<false>(inputs);
  UnpackedInput& unpacked_input = info_pair.first;
  InputFlags& input_info = info_pair.second;

  // Record input nodes if tracing
  // ...

  // Initialize backward function (and ctx)
  bool is_executable = input_info.is_executable;
  ctx->cdata.set_next_edges(std::move(input_info.next_edges));
  ctx->needs_input_grad = input_info.needs_input_grad.release();
  ctx->is_variable_input = std::move(input_info.is_variable_input);

  // Prepend ctx to input_tuple, in preparation for static method call
  auto num_args = PyTuple_GET_SIZE(inputs);
  THPObjectPtr ctx_input_tuple(PyTuple_New(num_args + 1));
  PyTuple_SET_ITEM(ctx_input_tuple.get(), 0, ctx_obj.release());
  // ...

  // Call forward
  THPObjectPtr tensor_outputs;
  {
    AutoGradMode grad_mode(false);
    THPObjectPtr forward_fn(PyObject_GetAttrString(cls, "forward"));
    if (!forward_fn) return nullptr;
    tensor_outputs = PyObject_CallObject(forward_fn, ctx_input_tuple);
    if (!tensor_outputs) return nullptr;
  }

  return process_outputs(cls, ctx, unpacked_input, inputs, std::move(tensor_outputs),
                         is_executable, node);
  END_HANDLE_TH_ERRORS
}

From here, we know that the cls.apply invokes cls.forward and prepares information for cls.backward. cls.apply takes its own class information and all parameters from Python. And those parameters will be applied on cls.forward. Noticabily, ctx object can carry the information to backward pass.

Now we can explore how DataParallel works.

Step 1 and Step 2: split minibatch on GPU:0 and move to GPU

scatter scatters the input argugments and returns a tuple of inputs and kwargs. The actual definiton is here:

def scatter_kwargs(inputs, kwargs, target_gpus, dim=0):
    r"""Scatter with support for kwargs dictionary"""
    inputs = scatter(inputs, target_gpus, dim) if inputs else []
    kwargs = scatter(kwargs, target_gpus, dim) if kwargs else []
    # ...
    return inputs, kwargs

scatter function is a recursive closure unwrap its input tensor(s). But its core is the Scatter class. This is a Function class which will operate differently in forward and backward.

class Scatter(Function):
    @staticmethod
    def forward(ctx, target_gpus, chunk_sizes, dim, input):
        # ...
        outputs = comm.scatter(input, target_gpus, chunk_sizes, ctx.dim, streams)
        # Synchronize with the copy stream
        if streams is not None:
            for i, output in enumerate(outputs):
                with torch.cuda.device(target_gpus[i]):
                    main_stream = torch.cuda.current_stream()
                    main_stream.wait_stream(streams[i])
                    output.record_stream(main_stream)
        return outputs

    @staticmethod
    def backward(ctx, *grad_output):
        return None, None, None, Gather.apply(ctx.input_device, ctx.dim, *grad_output)

Let’s focus on forward path. The backward path should be easier to understand. Thus on a multi-gpu environment, the input tensor will be sent to a PyTorch module for scattering and copying. This process needs synchronization.

The actual works in done by C++.

std::vector<at::Tensor> scatter(
    const at::Tensor& tensor,
    at::IntList devices,
    const c10::optional<std::vector<int64_t>>& chunk_sizes, // optional parameter is bad
    int64_t dim,
    const c10::optional<std::vector<c10::optional<at::cuda::CUDAStream>>>& streams) {
  std::vector<at::Tensor> chunks;
  if (chunk_sizes) {
    // ...
    // flatten tensor by size. The default uses the other path
    int64_t chunk_start = 0;
    for (size_t chunk = 0; chunk < chunk_sizes->size(); ++chunk) {
      const int64_t chunk_size = (*chunk_sizes)[chunk];
      AT_CHECK(chunk_size > 0, "Chunk size must be positive");
      chunks.push_back(tensor.narrow(dim, chunk_start, chunk_size));
      chunk_start += chunk_size;
    }
  } else {
    // usually a tensor is seperated into chunks among devices
    chunks = tensor.chunk(/*chunks=*/devices.size(), /*dim=*/dim);
  }
  at::cuda::OptionalCUDAStreamGuard cuda_guard;
  for (size_t chunk = 0; chunk < chunks.size(); ++chunk) {
    // ...
    // dispatch to different devices
    chunks[chunk] = chunks[chunk].contiguous().to(
        {at::DeviceType::CUDA, device_index}, /*non_blocking=*/true);
  }
  return chunks;
}

As a matter of fact, tensor would be split which is done by at::narrow in the end. Since the operation only happens to strides and sizes, the memory is reused! PyTorch takes zero copy seriously at every level. But an important insight is that tensor is splitted regardless of its shape. You need to align different input tensors by its total size instead of a particular dimension.

Step 3: copy models to GPU

replicate sounds easy! But no. So the module is passed in as network.

def replicate(network, devices, detach=False):
    from ._functions import Broadcast

    // ...
    num_replicas = len(devices)

    // copy model parameters
    params = list(network.parameters())
    param_indices = {param: idx for idx, param in enumerate(params)}
    param_copies = Broadcast.apply(devices, *params)
    if len(params) > 0:
        param_copies = [param_copies[i:i + len(params)]
                        for i in range(0, len(param_copies), len(params))]

    // copy model buffers
    buffers = list(network.buffers())
    buffer_indices = {buf: idx for idx, buf in enumerate(buffers)}
    buffer_copies = comm.broadcast_coalesced(buffers, devices)

    // copy model modules
    modules = list(network.modules())
    module_copies = [[] for device in devices]
    module_indices = {}

    // construrct modules
    // ...

    // copy every submodules
    for i, module in enumerate(modules):
        for key, child in module._modules.items():
            // ...
                    replica = module_copies[j][i]
                    replica._modules[key] = module_copies[j][module_idx]
        // and their parameters
        for key, param in module._parameters.items():
            // ...
                    replica = module_copies[j][i]
                    replica._parameters[key] = param_copies[j][param_idx].detach() \
                        if detach else param_copies[j][param_idx]
        // also buffers
        for key, buf in module._buffers.items():
            // ...
                    replica = module_copies[j][i]
                    replica._buffers[key] = buffer_copies[j][buffer_idx]

    return [module_copies[j][0] for j in range(num_replicas)]

The copying part to GPU is hard to be found. But we know they are Broadcast and comm.broadcast_coalesced(buffers, devices). Until this point, we have the model living in different GPUs.

Here is the Broadcast. The invocation starts with Broadcast.apply(devices, *params).

class Broadcast(Function):
    @staticmethod
    def forward(ctx, target_gpus, *inputs):
        // ...
        ctx.target_gpus = target_gpus
        if len(inputs) == 0:
            return tuple()
        ctx.num_inputs = len(inputs)
        ctx.input_device = inputs[0].get_device()
        outputs = comm.broadcast_coalesced(inputs, ctx.target_gpus)
        // ...
        return tuple([t for tensors in outputs for t in tensors])

    @staticmethod
    def backward(ctx, *grad_outputs):
        return (None,) + ReduceAddCoalesced.apply(ctx.input_device, ctx.num_inputs, *grad_outputs)

And its backward direction ReduceAddCoalesced looks similar.

class ReduceAddCoalesced(Function):

    @staticmethod
    def forward(ctx, destination, num_inputs, *grads):
        // ...
        grads = [grads[i:i + num_inputs]
                 for i in range(0, len(grads), num_inputs)]
        return comm.reduce_add_coalesced(grads, destination)

    @staticmethod
    def backward(ctx, *grad_outputs):
        return (None, None,) + Broadcast.apply(ctx.target_gpus, *grad_outputs)

So the core parts are comm.broadcast_coalesced and comm.reduce_add_coalesced.

The comm.broadcast_coalesced is implemented in C++. We have model parameters as tensors.

tensor_list2d broadcast_coalesced(TensorList tensors, IntArrayRef devices, size_t buffer_size) {
  if (!std::all_of(tensors.begin(), tensors.end(),
                   [&](const at::Tensor& t) { return t.get_device() == devices[0]; })) {
    throw std::runtime_error("all tensors must be on devices[0]");
  }
  // ...

  tensor_list2d outputs(devices.size());
  outputs[0] = tensors.vec();
  // ...

  for (auto & chunk : utils::take_tensors(tensors, buffer_size)) {
    // ...
    std::vector<at::Tensor> results;
    if (chunk.type().is_sparse()) {
      auto flat_tuple = utils::flatten_sparse_tensors(chunk.tensors);
      std::vector<at::Tensor> broadcast_indices = broadcast(flat_tuple.first, devices);
      std::vector<at::Tensor> broadcast_values = broadcast(flat_tuple.second, devices);
      // ...
    } else {
      std::vector<Tensor> results = broadcast(utils::flatten_dense_tensors(chunk.tensors),
                                              devices);
      // ...
    }
  }

  // ...
  return outputs;
}

This is too complicated and perhaps not insightful for our aim. But effecitively NCCL is used to broadcast tensor’s address to multiple GPUs. reduce_add_coalesced does the reverse.

Step 4: Forward pass

Finally, this step is obvious. But we need to pay attention to the output tuple because of the next step.

def parallel_apply(modules, inputs, kwargs_tup=None, devices=None):
    r"""Applies each `module` in :attr:`modules` in parallel on arguments
    contained in :attr:`inputs` (positional) and :attr:`kwargs_tup` (keyword)
    on each of :attr:`devices`.

    Args:
        modules (Module): modules to be parallelized
        inputs (tensor): inputs to the modules
        devices (list of int or torch.device): CUDA devices

    :attr:`modules`, :attr:`inputs`, :attr:`kwargs_tup` (if given), and
    :attr:`devices` (if given) should all have same length. Moreover, each
    element of :attr:`inputs` can either be a single object as the only argument
    to a module, or a collection of positional arguments.
    """
    # ...

    def _worker(i, module, input, kwargs, device=None):
        # ...
        try:
            with torch.cuda.device(device):
                # this also avoids accidental slicing of `input` if it is a Tensor
                if not isinstance(input, (list, tuple)):
                    input = (input,)
                output = module(*input, **kwargs)
            with lock:
                results[i] = output
        except Exception as e:
            with lock:
                results[i] = e

    if len(modules) > 1:
        threads = [threading.Thread(target=_worker,
                                    args=(i, module, input, kwargs, device))
                   for i, (module, input, kwargs, device) in
                   enumerate(zip(modules, inputs, kwargs_tup, devices))]

        for thread in threads:
            thread.start()
        for thread in threads:
            thread.join()
    else:
        _worker(0, modules[0], inputs[0], kwargs_tup[0], devices[0])

    # ...
    return outputs

results keeps track of results from every devices. Most cases, it will be a scalar loss tensor.

Python’s threading library is used to send things to models on different GPUs. This is a little bit expensive than I thought because of the overhead from threading.

Step 5: Compute

The gather has a similiar design as scatter. The core is Gather.apply(target_device, dim, *outputs). And its core is comm.gather(inputs, ctx.dim, ctx.target_device).

at::Tensor gather(
    at::TensorList tensors,
    int64_t dim,
    c10::optional<int32_t> destination_index) {
  # ...
  AT_CHECK(!tensors.empty(), "Expected at least one tensor to gather from");
  at::Tensor result;
  int64_t total_size = 0;
  auto& first = tensors.front();
  const auto first_size = first.sizes();
  std::vector<int64_t> expected_size(first_size.begin(), first_size.end());
  for (const auto& tensor : tensors) {
    // ...
    expected_size[dim] = tensor.size(dim);
    // ...
    total_size += tensor.size(dim);
  }
  expected_size[dim] = total_size;
  at::Device device(at::DeviceType::CPU);
  if (!destination_index || *destination_index != -1) {
    device = at::Device(at::DeviceType::CUDA, destination_index ? *destination_index : -1);
  }
  result = at::empty(expected_size, first.options().device(device));

  int64_t chunk_start = 0;
  for (const auto& tensor : tensors) {
    result.narrow(dim, chunk_start, tensor.size(dim))
        .copy_(tensor, /*non_blocking=*/true);
    chunk_start += tensor.size(dim);
  }
  return result;
}

It’s actually hard to find things. This basically compose a final tensor from tensors from devices. destination_index is the index of output device which is GPU:0 by default (defined as the parameter of data_parrallel).

Step 6: Loss value

This is the standard code you will write.

loss = model.forward(x)
loss = loss.sum()
optimizer.zero_grad()
loss.backward()
optimizer.step()

In next minibatch, same process will be sent to GPU.

Discussion

PyTorch focuses on abstraction and zero-copy. It’s more intutive to Tensorflow 1.x. It’s indeed an engineering success. Despite its simple interfaces, we need to understand its Tensor abstraction and those convenient helpers. Since CUDA uses different programming models, the work to hide CUDA details is enormous. PyTorch did this by mixing C++ and Python with pybind11. And it strives to maintain a similiar language in a large hierarchy that aligns the Tensor, Function, and Model. That’s remarkable.

Though, programming in a mixed languages are difficult. Type checking is always complicated. GIL needs to be remembered. With CUDA, synchronization, memory management and communication expands the scope of engineering work.

How to have a data science lab in 13 steps?

Thu, 20 Dec 2018 00:00:00 +0100

TL;DR. You have to be passionate and insane. I would not recommend doing it.

Research is an exciting thing. The purpose is to produce knowledge. You have the ultimate freedom because the job is about extending knowledge towards unknown.

1. Love Computer Science

You have to believe in what you are doing.

2. Learning Programming

Programming is a must. You have to write the implementation of your idea. And don’t forget the annoying part! Data processing will take the majority of your time, and there is no open source solution for that.

In some cases, you will need to use a lot of languages. I used C++/Java/Python/Ruby/Bash. Just learn.

3. Learning Mathematics

Things are useful: Linear Algebra, Probability, and Calculus. It’s for you. You don’t want to cry if you can’t even guess what theory belongs to from the paper. Knowing that you have to learn if you need something otherwise, you have to give up for your own thoughts.

4. Find good peers

Unless you have equipment and money. But wait a minute, it’s the least problem. If you don’t have a supervisor, you will have no way of knowing if you are actually thinking. Generally, more experience is better.

5. Find awesome papers

You are not alone. Papers are written by other miserable researchers. They tried to write it as well as possible. Welcome to this community. But you need to find good papers to start. Learn Scientific Method. If you failed on this, you can’t define your research problems.

6. Learn everything you need

Let’s read this paper in a day. Sorry, I mean it will usually be a week if you are not familiar with all the terms. Machine learning or deep learning, it doesn’t matter. Learn.

7. Be passionate and have some fun

Research is a lonely job. It’s one of the hardest jobs. Nobody can help you when you are creating. It requires hard work. And good motivation is from your position. And You need to rest. You need to get some good feelings from creation. Your brain is just not that good when it.

8. Believe it just takes time. A lot of time.

Learning takes time depending on how smart you are. I’m really dumb at getting all of that new knowledge. Programming takes time depending on how annoying the problem is. And your trials are part of the game. That takes weeks until you build a new mental schema.

9. Design your strategy and scientific problem

You need to focus on a tiny piece so you can work on even your research topic is fantastic. Keep looking until you have a good one.

10. Accept your ideas are shit

The secret is that it will get better if you think about it. Don’t stop.

11. Make a time plan

It’s not about catching the deadline. It’s for stopping. Can’t you do your research perfectly right? You have less data, limited time, time schedule, your future, and your health because you are exploring alone.

12. Treat people nicely

There are many researchers in the world working on the same thing. Write something that’s easier to understand so that you can also read a good paper. Reply to inquiries so that you can get a reply from the community.

13. Find more people

You can’t work alone when it’s starting to emerge with new directions and new ideas.

3 steps to start Google Summer of Code with Discourse

Sun, 02 Dec 2018 00:00:00 +0100

Discourse is a wonderful free and open source software for a modern forum. Now it has already powered many projects’ forum. Take a look at where it’s built and others.

Why Discourse?

Firstly, you need to know what project you are looking for. For example, what’s Discourse? Discourse is an evolving project with a client Ember.js app and a Rails backend. It focuses on people and community building. Discourse team is also professional and helpful. You can find them on Discourse Meta. Meta is the place where the team discusses how and what to develop Discourse. It’s transparent to all contributors.

They are also casual for your involvement. Remote work is how the team works. It’s a lot of freedom so does GSoC.

If you are interested, what’s next?

Secondly, how to prepare if you are interested in a project? There are two steps in this task. Read Google Summer of Code’s manual. Moreover, read how the Discourse team wants you to do. However, since you are reading my post, I’m going to tell you a few secrets. The first secret is Reading is # Fundamental. So let’s talk about GSoC manual firstly.

Read FAQ.
Read guides.
(Chinese) If you understand Chinese, there are more resources.
Then you can ask questions.

Before you continue on reading, let’s answer a question. Are you interested? GSoC aims to contribute open source community which means the aim is to make you comfortable and want to stay in the future. Hang on to that motivation otherwise. It’s going to be dull. My motivation would be that I hope Discourse becomes the largest platform for online discussion because I believe in the same value from the team. It will be useful if I need an online discussion forum.

Discourse team will be in favor of students that who have contributed to it. Also, it’s a straightforward process. Just start. Read contributing guidelines.

Take the smallest bite

It’s natural that feels overwhelming when it comes to a massive project. I stills feel the difficulty when I read the code. However, you should be able to find what is most interested. Find that smallest interests part. Analyze it. Write a plan. Also, execute your plan. If there’s a lion on the way, you can ask questions, and most likely you will get a reply quickly. As long as you know what you are trying to do, people understand you.

When it comes to proposal

Lastly, you need to apply. When you need to write a proposal, you feel it’s real now. That’s true. It would be better if you already have your plans and tried some ideas. As previously said, the team is helpful if you know them. Discourse team usually has a lot on their hands. If they know you, they can, of course, give you some tasks that you might be interested. Alternatively, better, they help you with your ideas.

Also, more importantly, consider your proposal as a guideline/todo list for yourself. So you don’t get lost.

Good luck for the incoming Google Summer of Code. Happy hacking!

Take it slow in life

Mon, 19 Nov 2018 00:00:00 +0100

Internet profoundly changes human life. I read a magazine when I was a kid. It says “Everything at your fingertip.” This refers to the internet. It finally becomes truth however terrifying reality. People start to live in the matrix and feed on their own biases. Nobody sells happiness. It can’t be traded anyway.

Being social is a fundamental need for a human. We used to work together for food and raise children. It’s hard-wired to our brain and defines feelings. But we are also mammals. Mammals look for variety. Internet and social apps give us this unbelieve variety. Without working on it, we get choices. Dating apps are a great example, swiping gives you dominating feelings and infinite choices. It works, and it works like a mirage. Entrepreneurs and engineers are working so hard to provide this options to us. People design the giant machine clusters running on their algorithm which made by few’s minds. They made it easy and fun. It’s great gamification. It’s highly addictive too. Though, this is detrimental too. Tinder uses ELO algorithm to ensure you are not going to see people outside your range (score). But life isn’t like this. Life has surprises. Meeting people brings enjoyment. Flirting requires skills and practices. And someone at the door might have the answer you are looking for your development. The habit swiping destroy the joy. It’s fake.

People don’t read

We are a species feed on information. We even have thinking on computer engineering that thinks information + context = knowledge. Knowledge is also written on language. We are incredible because we read. We learn from our greatest so that we don’t make the same mistakes. In fact, all technology comes with profound researches. Those are written over the years. We would die much easier without penicillin. We wouldn’t be able to predict weather without air balloon and mathematics. We wouldn’t be able to watch Netflix without decades of technological development. Yet people don’t read. We are starting to transit from learning from text to learning from images and videoes. Reading builds knowledge structure, but videoes only conveys feelings. Reading is fundamental. No matter who you are.

Why people don’t read?

II think learning is always a complicated process. Because you have to put long hours reading a subject which you don’t know. That feels uncomfortable. Maybe many people don’t have the epiphany moment. That brings you all the joy of learning. It’s generally true that you can learn whatever you want on the internet. Knowledge is accessisble than ever, but people are lack of motivation. Yes, learning might not lead you to a perfect job. It doesn’t give you a better income. But,

Difficulty is a process of mastering your craft. – Qunicy

We might have two different perspectives in looking of the world. Let’s start with hardest one.

Developing world

If people don’t have accessible internet, our assumption doesn’t make sense. People probably live based on their instincts or follow social codes. Try to imagine people living in the 1960s.

Then comes the Chinese. Rapid growth drives technological advancement and the internet. It builds a gigantic gap between rich and poor but internet puts them together. Widely accessible information without a privacy filter is terrible. Your need grows exponentially with the new information. But not your financials. Comparison between real life and life on photos draw pain and desire. No wonder Chinese are greedy in their relationships. The idea of better life requiring money is a illusion of not acknowledge time and development.

First World problem

Living in a developed world means knowledge and books are accessible. So do the internet and smartphones. It’s much more comfortable to live in bubbles. You only need to react when your life is at stake. Thanks to widely accessible higher education and good life quality. Living in the 20s are fun. Endless choices among other young people and the internet are always fun. It also develops comfortable numb. Being able to push yourself is the way to grow. And the only way to become is to work through the pain. Many people don’t go to find that drive. They are not living.

Happiness needs work

I love Yuval’s choice of Buddhist view.

If I identify happiness with fleeting pleasant sensations and crave to experience more and more of them, I have no choice but to pursue them constantly. When I finally get them, they quickly disappear, and because the mere memory of past pleasures will not satisfy me, I have to start all over again. Even if I continue this pursuit for decades, it will never bring me any lasting achievement; on the contrary, the more I crave these pleasant sensations, the more stressed and dissatisfied I will become. To attain real happiness, humans need to slow down the pursuit of pleasant sensations, not accelerate it.

Just like concrete work, it requires constant tweaking and coping. Life applies.

Digging in Python iterator and enumerate

Sun, 04 Nov 2018 00:00:00 +0100

When a PyTorch DataLoader repeat its data? Why it’s so magical and impossible to see in its code. It is implemented as an iterator in Python. Pythonic but implicit.

Use case

for i_batch, sample_batched in enumerate(data_loader):
    print(i_batch, sample_batched)

The enumerate is not hard to understand which only returns the sequence and iterator’s content. So what iterator does is the key to the puzzle.

Iterator

We can find Python’s document about the iterator:

An object representing a stream of data. Repeated calls to the iterator’s __next__() method (or passing it to the built-in function next()) return successive items in the stream. When no more data are available a StopIteration exception is raised instead. At this point, the iterator object is exhausted and any further calls to its __next__() method just raise StopIteration again. Iterators are required to have an __iter__() method that returns the iterator object itself so every iterator is also iterable and may be used in most places where other iterables are accepted. One notable exception is code which attempts multiple iteration passes. A container object (such as a list) produces a fresh new iterator each time you pass it to the iter() function or use it in a for loop. Attempting this with an iterator will just return the same exhausted iterator object used in the previous iteration pass, making it appear like an empty container.

The document says a lot. When you use next(data_loader), it gives you a new item. And StopIteration will be raised when it’s exhausted. When we use iter() and for loop with the iterator, a container object is built and as a fresh new iterator.

That sounds complicated. How does it allocate? Who manages the resources? What about processes? GIL?

And more puzzles come. The document points to itertor types. It didn’t explain the puzzles but introduced a new concept. A container can be used as an iterator if it supports __iter__(). Plus __next__(), it’ll form the iterator protocol.

But okay, it doesn’t explain anything. From my experience from Ruby (MRI), the implementation of Python should be similar to Ruby. Let’s dig in CPython.

CPython’s iterator support

CPython supports tp_iter as API. That’s a good start. If you need to support an API, you have to implement it. The search comes to Type Objects.

CPython seems to have a regular type object for every type. That’s good to find stuff. Maybe it’s also scary to have that long type definition if you have to define a new custom object. Loot at Asian’s work, Ruby (MRI) has a surprisingly clean interface since it’s managed in the object inheritance. We are too responsible for doing other’s work. Though, the document doesn’t explain how tp_iter and tp_iternext were expected to use.

They are two places defined them. One is included in typeobject.h and the other one is object.h. Everything is objected in Python, so this is more relevant for now.

typedef struct _typeobject {
    /* Iterators */
    getiterfunc tp_iter;
    iternextfunc tp_iternext;
} PyTypeObject;

But it doesn’t explain how different type implements it. A list is a common sense for Python programmer so let’s look at listobject.c. CPython uses reference counting GC. So heap space is used for object allocation. The other pairs of functions are reference counting.

List iterator object

typedef struct {
    PyObject_HEAD
    Py_ssize_t it_index;
    PyListObject *it_seq; /* Set to NULL when iterator is exhausted */
} listiterobject;

static PyObject *
list_iter(PyObject *seq)
{
    listiterobject *it;

    if (!PyList_Check(seq)) {
        PyErr_BadInternalCall();
        return NULL;
    }
    it = PyObject_GC_New(listiterobject, &PyListIter_Type);
    if (it == NULL)
        return NULL;
    it->it_index = 0;
    Py_INCREF(seq);
    it->it_seq = (PyListObject *)seq;
    _PyObject_GC_TRACK(it);
    return (PyObject *)it;
}

This finally explains how container objects get its allocator. A new struct and new object are used to get iterator in Python. Then we can come to an iterator object.

Iterator object

The funny thing is that you can’t construct an iterator in Python (iterator()). According to another document says:

An iterable object is an object that implements __iter__, which is expected to return an iterator object.

In the source code, you find similarity.

typedef struct {
    PyObject_HEAD
    Py_ssize_t it_index;
    PyObject *it_seq; /* Set to NULL when iterator is exhausted */
} seqiterobject;

PyObject *
PySeqIter_New(PyObject *seq)
{
    seqiterobject *it;

    if (!PySequence_Check(seq)) {
        PyErr_BadInternalCall();
        return NULL;
    }
    it = PyObject_GC_New(seqiterobject, &PySeqIter_Type);
    if (it == NULL)
        return NULL;
    it->it_index = 0;
    Py_INCREF(seq);
    it->it_seq = seq;
    _PyObject_GC_TRACK(it);
    return (PyObject *)it;
}


static PyObject *
iter_iternext(PyObject *iterator)
{
    ...

    result = PySequence_GetItem(seq, it->it_index);

    ...
}

It’s not much different except here it expects an iterable object. It also builds the iterator object. So the last thing would be where for allocates iterator.

How `for` allocates the iterator?

Let’s go back to the example for i_batch, sample_batched in enumerate(data_loader):. When for statements get evaluated, the testlist which is data_loader object if not consider enumerate will be evaluated. It’s evaluated by executing __iter__() and __next__(). And an implicit generator will get allocated!

Until now, we know how CPython allocates the iterator as well as who manages the resource.

Process management and GIL?

It’s a wrong question as of now. If CPython manages the resource, it’s always going to be local and constrained by GIL.