The Main Part That Stores Information Within A Computer Is The Hard Drive.
Besides the keyboard; tapes disks scanners modem punched cards mouse (or equivalents)
It depends on what kind of info you mean, to record a voice or noise use a voice recorder but if you want to store computer info use a usb memory stick.
Primary Device for storing information on the computer would be a hard drive. Temporary Storage would be RAM (Random Access Memory)
Flash-drives, or "thumb-drives"can hold digitized information, along with External hard drives and other things.