Every engine has limits which determine the peak number of revolutions per minute that it can achieve. There are many parts which become more stressed as engine speed increases, and the part that would be the first to fail on whichever engine you are working on is the first part you should consider upgrading if you wish to increase an engine's peak RPM. However, most manufacturers don't build engines with a lot of parts which would out-perform other parts by a considerable amount, so raising an engine's RPM often requires a number of upgrades instead of just one part.
Camshaft
The most common limitation in piston engines is the valves. Because of this, the camshaft, which controls when the valves open and close, is often the main focus when it comes to RPM peaks. Since the valves have to open, let air in or out, and close before the piston comes back to TDC, a cam lobe which allows the valve to close sooner will allow the engine to reach higher RPM levels before any piston/valve collisions occur.
Valve Springs
Valve springs are the second valve-related component which is likely to limit high-RPM functionality. These springs are what force the valve to return to its seated position. Stiffer springs are harder for the camshaft to compress, which consumes engine power, so high-performance engines often have finely-tuned valves which are only stiff enough to allow the desired RPM to be reached. This is often the case for production engines, too, since horsepower numbers typically sell more cars than peak RPM numbers.
Cooling
More RPM means parts are moving faster, causing more friction, which generates heat. Hot metal is more likely to break, so an engine's peak RPM limit may be due to heat issues. If this is the case, upgrading the engine's cooling system is likely necessary. At this point, however, you put more stress on the cooling system and oil to work perfectly in order to prevent engine failure. Assuming the cooling system isn't going to work at peak efficiency and designing for that is a good practice.
Some points of the engine are nearly impossible to cool effectively, such as valves and pistons, so they may become the peak limitation at the end of the project.