Pretty much. I set one up for the first time a couple weeks ago and it could not have been easier. Bare minimum "hello world" would basically just be a board, a USB cable, a computer with a USB port, and an LED.
As FYI (you may already know this @morganherlocker, so this is for the n00bs) there is a controllable LED built into the UNO (and similar boards) attached to pin 13, so this is possible without any additional parts. In addition to LEDs and a breadboard, some push buttons, and maybe a photoresitor or a rotary potentiometer to play around with analog inputs are all good starting items.