Tiny Corp’s $30,000 tinybox Red Kimi targets local runs of a 2.8-trillion-parameter AI model - Hardware, PCs and semiconductors

The workstation combines four AMD Radeon AI Pro R9700 GPUs with 16 TB of NVMe storage. Tiny Corp claims it can run Moonshot AI’s Kimi K3 at 40 tokens per second, but has not published the test conditions.

Tiny Corp’s new tinybox Red Kimi is a $30,000 workstation designed to run Moonshot AI’s 2.8-trillion-parameter Kimi K3 model locally. Its hardware pairs four AMD Radeon AI Pro R9700 graphics cards with sixteen 1 TB NVMe SSDs, using fast storage to accommodate a model too large to fit in the GPUs’ memory alone.

The four GPUs provide 128 GB of GDDR6 memory in total, while the SSD array offers 16 TB of capacity and a claimed 238.4 GB/s of bandwidth. Tiny Corp says its tinygrad software includes a driver that allows the GPUs to access the NVMe drives directly instead of routing every read through the CPU. The system also has a 32-core AMD Epyc Genoa processor and 32 GB of DDR5 system memory.

Tiny Corp claims the machine can generate Kimi K3 at 40 tokens per second. That figure has not been independently tested, and the company has not disclosed the precision, context length, batch size or method used to measure it.

The performance claim depends on keeping much of the model’s frequently used data in GPU memory. Kimi K3’s published checkpoint is around 1.56 TB, far beyond the tinybox’s 128 GB of GPU memory. The model activates 104 billion parameters for each token, which would require reading at least 55 GB of weights per token at the stated effective precision. The article’s analysis estimates that reading all those weights from the SSD array would support only about four tokens per second.

To reach 40 tokens per second, the system would need to draw heavily on GPU memory, with the SSDs supplying only part of the data required for each token. The result could depend on how well the system caches weights, as well as on factors such as quantization or speculative decoding; Tiny Corp has not explained which approach it uses.

The Red Kimi is built on Tiny Corp’s G5 platform. Its configurator estimates a weight of about 56 pounds and dimensions of 12 inches wide, 19.25 inches tall and 18 inches deep. Tiny Corp also lists noise below 60 dB for configurations with the R9700 cards, noting that their blower fans are audible.

At $30,000, the system is priced close to an Apple Mac Studio configuration that starts at $22,299 and reaches $29,799 with 16 TB of storage, according to the source report. The Red Kimi’s appeal is its use of SSD capacity to run a model locally, but its claimed generation speed remains unverified outside Tiny Corp.

Sources: TechRadar.