He is introduced in Reg'lar Fellers Heroic Comics #15 (November 1942). Introduced is a strong word, because The Music Master just picks up the phone and asks him to come over and help him fight Gurkee, the mad scientist of sound, like they've known each other for a while. The next issue, #16 (January 1943) contains the story "The Demented Apparition" which has them meeting and teaming up. It appears the stories were printed out of order.
Penrod is the heir to the Hunter fortune. His grandfather John built the Mosque Theater in Gotham. His uncle John Cranston was his guardian, but disappeared. He pretended to haunt the theater to scare people off so he could search it for John Hunter's fortune. When Penrod wanted to donate the theater to the Red Cross, Cranston had to start murdering people, apparently. Cranston dies at the end of the story.