Using Task Scheduler, one way to run a task more often than once a day was to repeat the task within the day. For instance, if you wanted to run a task every hour, you would repeat the task once an hour for that day. Now, if the task's running time is longer (e.g. the task is delayed by a slow network connection) than the repeat interval then another instance would be started even if you specify Do not start a new instance!
For a backup task, my workaround is to specify an hourly repeat interval and call a wrapper (e.g. a CMD or VBS script) to test for a lock file before executing the main program. In the wrapper, if the lock file exists then the wrapper aborts else it creates a lock file, runs the main program then deletes the lock file.
It looks like a task's Do not start a new instance parameter only applies to triggers. With Task Scheduler 2.0 (Vista onwards), you can create triggers for specific times of the day (say 01:00, 02:00, etc. for hourly triggers) and only one instance of a task is started. It's a pity there aren't shorter trigger intervals because to schedule a task every hour of the day, you have to create 24 triggers.