The PS3 licked the Many cookie

This post is inspired by
I want a
good parallel computer

It is important to understand why the
PS3 failed. The perspective here was juniorish dev working on simulation and
rendering in triple-A. I remember less than I knew and knew less than most!

However, what I can provide here is the hindsight of someone who actually developed and shipped titles
¹ on
the PS3.
I wanted the PS3 to succeed. To be more specific; I wanted Many-Core to succeed.

The PS3 failed developers because it was an excessively heterogenous computer;

and low level heterogeneous compute resists composability.²

More like Multicore than Many

The primary characteristic of Many-Core is, by virtue of the name, the high core count. Many-core is simply a tradeoff
that enables wide parallelism through more explicit (programmer) control.

At first glance, the SPEs of the PS3 fit the bill. They seem to have all the characteristics of Many-Core.
The
problem is that most important characteristic, that there is many cores, is significantly lacking.

First off you didnt get the full 8 SPEs as a (game) developer. Out of the 8 SPEs one was disabled due to die yield
and the
OS got a core and a half. While this changed with updates one only really got 5-6 SPEs to work with. The Xbox360
in
contrast had what amounted to 3 PPEs (2 more). So the Cell really featured at most 3 more (difficult to use) cores
than the Xbox360.

Computationally Weak Components

The claim from wiki is a single SPE has 25 GFlops and the PS3 GPU has 192 GFlops. If
you
absolutely maxed out your SPE
usage you would still not even be close to the power of the underpowered PS3 GPU. For contrast the Xbox360
gpu had
240
GFlops. The GPU of the PS3 has separate Vertex and Pixel shading. In contrast, the XBox360 was shared
computational resources so it could load balance between heavy vertex shading vs heavy pixel shading. (Examples
here would be character skinning
vs UI rendering)

As a game developer these Flops numbers reflect the experience of developing on these respective platforms. This
was particularly noticeable in something like post processing where the demands of the vertex unit are very low
(large quads).

Due to the weakness of the GPU vertex unit developers would use the SPEs to do skinning.
The pixel shading unit did not have constants. So one would also have to do shader patching on the SPEs before
these
programs could be sent to the GPU. All of these things require synchronization between the CPU, SPE and GPU and
interact with workload balancing. In retrospect I also assume that dynamic branching in the shader was either
impossible or prohibitive so this is why everyone did excessive shader permutations. This means 10s of megabytes
of shaders. Again contrast this with the XBOX360 which supported wave
operations ³ and I even used this feature back in the day. Because each component of the PS3 is
weak on its own they all must be employed in concert to compete with (the) less heterogeneous platforms.

Computer Not Super

While the Cell could behave more like a supercomputer I saw it mostly used more like generic GPU compute. I never
saw production code that did anything but dispatch N jobs from the PPE. I never saw direct inter SPE communication
even though I recall such a thing was possible (mailboxes). This is similar to how GPU inter workgroup workloads
are more rare and difficult.

The hetrogenous nature was everywhere. Even the PPE was quite different from an SPE. The SPEs had only vector
registers; the PPE had fp, gp, and vector
registers. Is this really bad? No⁴, but it makes everything more heterogeneous and therefore more
complex. Getting
maximum performance out of these SPE units means that you were likely doing async DMAs while also doing compute
work. These nuances could be a fun challenge for a top programmer but ends of being more of an obstacle to
development for game studios.

Sharp Edges

The PS3 had 512 Mb total memory but 256 MB was dedicated to graphics and only had REDACTED Mb/s access from the
CPU. So
this means in addition to the 256 MB purely for graphics you would also have to dedicate system memory for
anything that was written to and from the GPU. The point here is inflexibility and heterogeneous nature.

The PS3 had cool features but these were never general purpose and could only be exploited by careful attention to
detail and sometimes significant engine changes. I recall using depth bounds for screen space shadowing and
one could probably use it for a few other similar gpu techniques (lighting). There was also the alluring double z
writes which is a one-off for depth maps if you dont actually use a pixel shader. I don’t recall all the edges,
but they were sharp and it meant performance cliffs if one strayed off them. The next section covers the sharpest
edge of them all.

The Challenge of Local memory

Of course the challenge that everyone knows about the SPEs is the constraint of memory access to local memory. You
got 256Kb but in reality once you factored in stack and program you were probably down to 128Kb. This
computational model is far more restrictive than even modern GPU compute where at least there you can access
storage buffers directly.

Most code and algorithms cannot be trivially ported to the SPE.
C++ virtual functions and methods will not work out of the box. C++ encourages dynamic allocation of objects but
these can point to anywhere in main memory. You would
need to map pointer addresses from PPE to SPE to even attempt running a normal c++ program on the SPE. Also null
(address 0x0) points to the start of local memory and is not a segfault to load from it.

So, instead of running generic code on the SPE, what developers did was write handcrafted SPE friendly code for
heavy
but parallelizable parts of their engine. With enough talent and investment you can eke out the full compute
power of the PS3.⁵ Of course this is maybe easier as a first party developer as you
can at least
focus on this exotic hardware and craft your engine and game features around the type compute available. This is
why Naughty Dog
famously came so close to showing us the full potential of the console.

What could have been

Had the PS3 been what was originally designed it would have been a much more exotic but much less heterogeneous
machine.

The original design was approximately 4 Cell processors with high frequencies. Perhaps massaging this design would
have led to very homogenous high performance Many-Core architecture. At more than 1 TFlop of general purpose
compute it would have been a beast and not a gnarly beast but a sleek smooth uniform tiger.

One has to actually look at the PS3 as the
licked cookie
⁶ of
Many-Core designs. This half-baked, half-hearted attempt became synonymous with the failure of Many-Core.
I
used to think that PS3 set back Many-Core for decades, now I wonder if it simply killed it forever.

Refs

¹ Some of
the titles I have worked on.

² I will not provide a proof or reference but the mathematical issue is that the space is not covered
uniformly. This low level composability problem is often seen instruction selection when writing assembly.

³ The XBox360 gpu had wave operations like ifany/ifall that are similar to modern control flow
subgroup operations.

⁴ The fact that it was only vectors on the SPEs was present to the programmer due to loads/stores
having to also be vector aligned

⁵ PS3
SPE usage : it is clear that some games had higher utilization than others.

⁶ I am not sure my usage fits Raymond’s narrow specifications.

classic.copetti.org
Source for images and some technical specifications.