The word "attention" originates from "ad" (toward) and the verb "tendere" (to stretch, extend, or aim), forming the word attentio in Latin. "Attentio" means focus something. At its core, attention requires effort, an active stretching of mental resources. Its nature, however, tends toward transience. Even staring at a fixed point on a wall becomes a challenging exercise in maintaining focus.

As Hermann von Helmholtz aptly describes:

“The natural tendency of attention when left to itself is to wander to ever new things; and so soon as the interest of its object is over, so soon as nothing new is to be noticed there, it passes, in spite of our will, to something else. If we wish to keep it upon one and the same object, we must seek constantly to find out something new about the latter, especially if other powerful impressions are attracting us away.”

Strain of Attention

One of the main causes of the difficulty in focusing in the current world is an increase of information. Although multitasking is frequently cited as a solution, Earl Miller of MIT contends that this is untrue:

“Effective multitasking is a MYTH!”

The human brain can process no more than 3–4 things at once. However, pushing these limits creates strain on attention, cause to mental fatigue and stress.

The process of requiring focus requires depth. However, people often prefer to fool themselves by thinking they can handle more than they actually are. Depth is frequently overlooked in today’s fast-paced world, where limited time favors superficial thought over deeper reflection.

Self Attention

The self-attention mechanism presented in the paper “Attention Is All You Need.¹” is among the most important advances in artificial intelligence. This approach, which is made possible by the Transformer design, simulates the attention mechanism by determining the relative relevance of each piece of data.

The algorithm evaluates relationships between words in a sequence and determines their relative importance. For example, in the sentence “She gave the book to him because it was his birthday”, self-attention helps the model understand that “it” refers to “the book” and “his” refers to “him.”

This procedure is similar to how the human attention system works. As Hermann von Helmholtz described, sustaining focus on a particular object requires continuous effort. In Transformer models, this effort is realized through the constant interaction of each unit within the context with other units.

Attention, whether human or machine, is a fleeting and effortful process. In a world inundated with distractions, the strain of attention highlights the necessity of intentional focus.

---

1. The paper Attention Is All You Need by Vaswani et al. (2017) introduced the Transformer architecture, a groundbreaking model in deep learning, particularly for natural language processing (NLP).