It turns out you can't upgrade your PS3 HDD without an original Dualshock 3 controller, which I never had.
System just goes into "safe mode", and won't accept any input from a third-party controller or Dualshock 4.
The not so funny part is that even if you put the original HDD back, the system still won't go out of the safe mode.
But, if you pull it out, let the system complain that no HDD was detected, and then put the original HDD back in without turning off the system - it suddenly works!

Consoles are fun, they said. No hassle upgrading, they said.