Early Denver grew in a north-easterly direction. One factor luring people toward Cole-Whittier was Ford Park, a private park/race track laid out in the 1860s between Downing and Race Streets and 36th - 38th Avenues. Then out in the country, Ford Park was an early Denver recreation area. It was supplanted by the coming of the railroad in 1870 when first the Denver Pacific and then the Kansas Pacific entered Denver near 40th Avenue and Downing Street. In fact, the northern border of the neighborhood along 40th Avenue was the location of the original Kansas Pacific tracks.

With the railroad came railroad workers. Many lived near the tracks east of Downing Street where the railroad yards were located. Annunciation Church at 36th Avenue and Lafayette Street was originally an Irish railroad workers' church. Similarly, the forerunner of St. Anthony's Hospital began in 1884 as a Union Pacific Hospital for railroad workers near 40th Avenue and York Street.

Fuller Park at 28th Avenue and Gilpin Street, Denver's second oldest park, was given to the city in the late 1870s. The developers were blunt about their intentions: by having the park as an amenity of the new area, people would make a point of moving close to it, thereby enhancing real estate values. This generally worked since, by the 1880s, the land around Fuller Park was one of the most prominent and flourishing areas of the Mile High City.

The Cole neighborhood, which became part of the city in 1874, stretches from 32nd Avenue to 40th Avenue and from York Street to Downing Street. Cole Junior High School and the neighborhood itself are named after Carlos M. Cole, a superintendent of Denver's Public Schools who was instrumental in establishing junior high schools in Denver. More than half of the residential blocks were developed prior to 1900. Several historic buildings remain, like the cleanly designed red brick and sandstone Wyatt School on 3620 Franklin Street, built in 1887 and named after its former principal George W. Wyatt.

Working class residents always set the pace for the neighborhood. The presence of Manual High School near Fuller Park is indicative of this. Manual was originally established as a branch of East High School in the 1890s. Students were then given the option of either attending East proper and receiving a vigorous classical education, or attending the Manual Training High School and learning a vocation. Today the two schools are known as the East/Manual High School Complex. Students may choose to go to either or both schools.