List of non vegan colors and dyes

BETA CAROTENE: Provitamin A. Carotene. Found in many animal tissues an in all plants. Used as a coloring in cosmetics and in the manufacture of Vitamin A. Question is if it really has animal source in any cases?

CARMINE: Cochineal. Carminic Acid. E120. Red pigment from the crushed female cochineal insect. Reportedly 70,000 beetles may be killed to produce one pound of this red dye. Used in cosmetics, shampoos, red apple sauce and other foods. May cause allergic reactions. Alternatives: beet juice, no known toxicity (used in powders, rouges, shampoos); alkanet root, from the root of an herblike tree, no known toxicity (used as a red dye for inks, wines, lip balms, etc. and can be combined to make a copper or blue coloring). Natural Red 4 (Red#4) also known as crimson lake, carmine, carminic acid, or cochineal. The dye is made from an insect called the cochineal, which lives on cacti in Mexico and South America, and is neither vegetarian, kosher, nor halal.

SILK POWDER: Obtained from the secretion of the silkworm. Used as a coloring agent in face powders, soaps, etc. Causes severe allergic reactions; systemic reactions if inhaled or ingested.

Some color that might have animal source:
E101 (B): Riboflavin, Vitamin B2
E101a (B): Riboflavin-5-Phosphate
E104 (B): Quinoline Yellow
E150a (B): plain caramel. It is manufactured by heating carbohydrates with or without acids or alkalis
E150b (B): caustic caramel. It is produced like E150a, but in the presence of sulfite compounds
E150c (B): ammonia caramel. It is produced like E150a, but in the presence of ammonium compounds
E150d (B): sulfite ammonia caramel. It is produced like E150a, but with both sulfite and ammonium compounds
E153 (B): Carbon Black, Vegetable Carbon
E160a (B): Alpha-carotene, Beta-carotene, Gamma-carotene. Can be made synthetically, but usually extracted from carrots. Gelatine may be used as a stabiliser
E170 (B): Calcium Carbonate, Chalk

Some artificial colors are made with animal ingredients. But I have not found any information on what colors that would be.

More non vegan ingredients can be found here.

A website with everything about food colors is found here.

Vegan Society has more information as well with Animal-Derived Possibly Animal-Derived Additives and more.

A list of food additives and food coloring and E-numbers on Wikipedia.