Born in Oakland, California to a musical family from New Orleans. His mother was a pianist and his father was a violinist fluent in both jazz and classical repertoires. The young Colescott developed an interest in art, which was encouraged by a family friend, an African American sculptor associated with the Harlem Renaissance named Sargent Claude Johnson.
Drafted into the Army in 1942, Colescott found himself in Paris at the end of World War II. Upon leaving the service, he received a BFA from UC Berkeley in 1949 (where he also earned an MFA in 1952), before returning to the City of Light for a year of study with the modernist Fernand Léger.